Based on Andy's comment, I am withdrawing this PR and submitting pieces of work as separate prs. I will document that list of new PRs as it is submitted.

Logger function added to recorder. This is no longer a prototype. However, it functionally complete now but has some issues (see Not Done).

I am keeping this WIP for another day or so while I work out the not done list below.

Includes:

1. New LogOperationRecorder class in recorder logs operations request/responses and http request/responses. Note that logging for each can be defined separately so that each can have a destination (file, stderr), log_level(debug, etc) and detail level (how much information is output)
2. Addition of PywbemLoggers class in _logging that manages loggers creating new named loggers and keeping the information that logging cannot handle (detail level). Includes parsing of the command line parameter for logging so wbemcli and pywbemtools do not have to do this separately.
3. Add new script test for wbemcli (test_wbemcli.py). Only includes a few tests today but new tests can be easily added.
4. Extend wbemconnection to add new constructor (enable_logging) that is a boolean. When this is created, the LogOperationRecorder gets the detail level info from PywbemLoggers by logger name.

Not done:

1. The tests in test_logging and test_wbemcli are incomplete. COMMENT: Move any new tests forward to new issue
2. Some of the new code in _logging.py is a hack and just needs refactoring. In fact, I have only lightly tested the final result. Cleaned up. Done
3. There is no documentation on the logging (or for that matter the statistics) in the documentation. COMMENT: Create new issue
4. I left the input parameters in their original order for the moment (destination:detail:log_level rather than Andy's proposal for different order. We should discuss the levels of detail (min, all) right now and exactly what they mean. The structure is in place so we can manage this detail level as we want to. Right now detail output 1000 char and all outputs everything. Maybe a hdrs_only or something but not sure exactly what that means. COMMENT: Create new issue
5. Not displaying enough in wbemcli when logger enabled. Just says logging=true no info about components, etc. Minor. COMMENT: Create new issue
6. Only outputs logger or test recorder, not both. I know how to extend to do both but wanted to get this much running first. COMMENT: Create new issue
7. I need to review logging of the wbemconnection itself since it is NOT linked to the other log outputs. The logical way to do this might to be to set a counter that increments for each new connection and this counter also is set into the ops and http logs. COMMENT: Create new issue

DISCUSSION:

1. Do we really need to be able to do both log and test recorder at the same time.
2. Do we need to link the ops/http logs to the connection.

This is an initial stab at addressing issue #66.

Please review and post comments here.

The docs for this listener branch are built automatically on RTD, so that the WBEM Listener API can better be reviewed:

https://pywbem.readthedocs.org/en/listener

Note there are now two new sections:

• WBEM server API
• WBEM listener API

In addition to the documentation, there is now a tool that demonstrates how to use the WBEM server API: tools/explore.py. Invoke without arguments for help.

I understand the desire to make internal code functionality into a separate library, but it comes at a cost. For distributions like fedora that package each python library as a separate package, it requires a maintainer to create a new package, get the package through the approval process, and then maintain/test it over time.

At the moment those that are involved in pywbem packaging on fedora are considering dropping/orphaning pywbem in fedora as the packaging burden has increased (unless we can find someone to take it and all the new dependencies over). If this occurs a few other packages in fedora which utilize pywbem will get dropped/orphaned too.

Is anyone here willing to help get these new packages specific to pywbem packaged for fedora and other distributions?

Maybe pywbem could come up with some other alternative, like git submodule? Keep this needed functionality together and separate for reuse? I'm not familiar with submodules and how difficult that would be do integrated releases in github for example.

Suggestions/help welcomed

Thanks

Ref. https://bugzilla.redhat.com/show_bug.cgi?id=1868128

Initial implementation of pywbem_mock.

This implements mocks of all of the WBEMConnection methods that contact the wbem server with a local repository that responds to the requests with data from the repository.

This is not a complete implementation of a WBEMserver and therefore creates a number of simplifications that are documented in the documentation page

Status Sat 6 jan 2018:

Note that this fixes the test failure from Friday. That was bad code compounded by fact that each python version has different ordering for dictionaries. V 3.4 ordering caught my instance properties error that depended on property ordering by accident.

1. Implementation complete and coverage is up to about 90%. There are a number of tests to add but we know that the paths generally work. The only components that I really need to work on to assure quality are . a. EnumerateInstance IncludeQualifiers, property lists, and DeepInheritance (tested but not all of the possible combinations), b. ModifyInstance (complex and may not have tested all options), c. c. c.Associations (am I picking up all of the possible options/variations), d. Do the open/pulls do exactly the same thing as the original operations.

2. ModifyClass not implemented.

3. About 600 tests now and 89% coverage of code. Still may be some error conditions not tested.

Removed WIP status and as far as I am concerned, lets commit this first version and do fixes with issues.

This also adds a diagram. I am not married to this diagram and suggestions welcome

It extends the description, etc.

NOTE: I thought I had set up th pr before but the branch was there but no pr. Not sure why

Removed the README.md

Please Review COMMENT 18 aug: KS WIP now because run_cimoperations.py needs update to match last changes. Only changes for subscription and listener code is a) commments, b) run_cimoperaitons test fail.

Breaks listener and subscription manager apart. Adds at least some support for static filters, subscriptions.

Discussion: Known issues:

1. I used two different patterns for the add functions and remove functions. In the add_filters i added an optional attribute to define if the filter is dynamic or static. In the remove indications and filters I modified so we have two different functions remove_static_filter, and remove_dynamic_filter. That creates inconsistency. DISCUSION: What is the most logical pattern for creates and removals now that we really will support static and dynamic filters. Thus, for example

     sub_mgr.add_filter(...dynamic_filter=false)
     sub_mgr.remove_static_filter(...)

1. When creating a static filter, it always creates a new destination. That is probably OK for the moment but not efficient if we are going to do may filters.
2. Right now I have the indication_id attribute in both the listener and subscription manager. I need to sort out where it really should be. Is this an id for the listener or for the subscription manager. DISCUSSION: Still thinking that one out and not sure if this is really a listener thing or subscription_manager. Note that we really only have one listener for subscription manager at this point.
3. The use of static is different here than in dsp docs. Our definition is a filter, subscripiton that is not controlled iwhtin the life cycle of a subscription manager instance. The subscripition manager may create and delete static subscriptions but there may also be real static subscriptions in an implementation which cannot ever be deleted. DISCUSSION: Should we be using a different name
4. the code does not exercise any control over static subscriptions mixed with dynamic filters so far. Should really add a test so that a subsubscription must be static if the filter is not in the dynamic_filter list, etc.
5. We are not marking static vs dynamic in the name component, etc. for filters and subscriptions. The difference is that dynamics are in the lists maintained in the subscription_manager. But there is no function such is is_dynamic_filter. That is just a comment now.

Passes tests with the examples code and have not yet run the run_cimoperation.py test. I have not created tests for manual filters, subscriptions.

This is still a work in progress for today but functionally complete. I need to run tests and we may want to refactor a bit. Note that I am inconsistent between create/remove, etc. on whether the static vs dynamic is a attribute of the call or separate function.

I'm new to PyWBEM and got this issue while executing this script

(tools) D:\Soft\PyWBEM\pywbemtools\pywbemtools>workon tools (tools) D:\Soft\PyWBEM\pywbemtools\pywbemtools>make develop File "", line 1 import sys; sys.stdout.write('s.sys.version_info[0:3]) ^ SyntaxError: EOL while scanning string literal File "", line 1 import sys; sys.stdout.write('sys.version_info[0]) ^ SyntaxError: EOL while scanning string literal Traceback (most recent call last): File "", line 1, in TypeError: 'str' object is not callable File "", line 1 $(printf 'try:\n from pbr.version import VersionInfo\nexcept ImportError:\n pass\nelse:\n print(VersionInfo(\042pywbemtools\042).release_string())\n') ^ SyntaxError: invalid syntax File "", line 1 $(printf 'try:\n from pbr.version import VersionInfo\nexcept ImportError:\n pass\nelse:\n print(VersionInfo(\042pywbemtools\042).release_string())\n') ^ SyntaxError: invalid syntax File "", line 1 $(printf 'try:\n from pbr.version import VersionInfo\nexcept ImportError:\n pass\nelse:\n print(VersionInfo(\042pywbemtools\042).release_string())\n') ^ SyntaxError: invalid syntax File "", line 1 $(printf 'try:\n from pbr.version import VersionInfo\nexcept ImportError:\n pass\nelse:\n print(VersionInfo(\042pywbemtools\042).release_string())\n') ^ SyntaxError: invalid syntax File "", line 1 $(printf 'try:\n from pbr.version import VersionInfo\nexcept ImportError:\n pass\nelse:\n print(VersionInfo(\042pywbemtools\042).release_string())\n') ^ SyntaxError: invalid syntax File "", line 1 $(printf 'try:\n from pbr.version import VersionInfo\nexcept ImportError:\n pass\nelse:\n print(VersionInfo(\042pywbemtools\042).release_string())\n') ^ SyntaxError: invalid syntax File "", line 1 $(printf 'try:\n from pbr.version import VersionInfo\nexcept ImportError:\n pass\nelse:\n print(VersionInfo(\042pywbemtools\042).release_string())\n') ^ SyntaxError: invalid syntax File "", line 1 $(printf 'try:\n from pbr.version import VersionInfo\nexcept ImportError:\n pass\nelse:\n print(VersionInfo(\042pywbemtools\042).release_string())\n') ^ SyntaxError: invalid syntax File "", line 1 $(printf 'try:\n from pbr.version import VersionInfo\nexcept ImportError:\n pass\nelse:\n print(VersionInfo(\042pywbemtools\042).release_string())\n') ^ SyntaxError: invalid syntax File "", line 1 $(printf 'try:\n from pbr.version import VersionInfo\nexcept ImportError:\n pass\nelse:\n print(VersionInfo(\042pywbemtools\042).release_string())\n') ^ SyntaxError: invalid syntax wget -q http://pywbem.readthedocs.io/en/latest/_downloads/pywbem_os_setup.sh process_begin: CreateProcess((null), wget -q http://pywbem.readthedocs.io/en/latest/_downloads/pywbem_os_setup.sh, ...) failed. make (e=2): The system cannot find the file specified. make: *** [pywbem_os_setup.sh] Error 2

(tools) D:\Soft\PyWBEM\pywbemtools\pywbemtools>https://github.com/pywbem/pywbemtools.githttps://github.com/pywbem/pywbemtools.git

Keeps time statistics on cim_operations. This keeps statistics on time to execute and request/response sizes for the cim operations.

Implements a Statistics class and connects cim_operations to that class to record execution time, request size, and reply size in that class for each operation type. It saves count, avg time, request size, response size in and exception count in that class for each WBEM operation type by name.

The Statistics class provides for snapshotting and resetting the statistics in addition to the functions to record each new statistics with start_time() and stop_time()

NOTE: At this point the statistics are not really general. The class OperationStatistics specializes the statistics for the operations and their is no provision for another class with its specialization at this point.

Finally, includes ability to disable statistics but I am not really using that capability. You enable statistics for each WBEMConnection.

In addition, the change adds three attributes to WBEMConnection.

last_req_len - Always last_reply_len - Always last_operation_time - Only when the enable-stats is set

ANDY: I goofed and force updated over your changes before I realized you had made them.

NOTE: This does NOT capture info from the server statistics.

• Errors: Got error when run python setup.py from release tarball from (https://github.com/pywbem/pywbem/releases/tag/0.12.2)

ERROR:root:Error parsing
Traceback (most recent call last):
  File "/home/fge/Downloads/pywbem-0.12.2/.eggs/pbr-4.0.2-py3.6.egg/pbr/core.py", line 96, in pbr
    attrs = util.cfg_to_args(path, dist.script_args)
  File "/home/fge/Downloads/pywbem-0.12.2/.eggs/pbr-4.0.2-py3.6.egg/pbr/util.py", line 258, in cfg_to_args
    pbr.hooks.setup_hook(config)
  File "/home/fge/Downloads/pywbem-0.12.2/.eggs/pbr-4.0.2-py3.6.egg/pbr/hooks/__init__.py", line 25, in setup_hook
    metadata_config.run()
  File "/home/fge/Downloads/pywbem-0.12.2/.eggs/pbr-4.0.2-py3.6.egg/pbr/hooks/base.py", line 27, in run
    self.hook()
  File "/home/fge/Downloads/pywbem-0.12.2/.eggs/pbr-4.0.2-py3.6.egg/pbr/hooks/metadata.py", line 26, in hook
    self.config['name'], self.config.get('version', None))
  File "/home/fge/Downloads/pywbem-0.12.2/.eggs/pbr-4.0.2-py3.6.egg/pbr/packaging.py", line 816, in get_version
    name=package_name))
Exception: Versioning for this project requires either an sdist tarball, or
access to an upstream git repository. It's also possible that there is a
mismatch between the package name in setup.cfg and the argument given to
pbr.version.VersionInfo. Project name pywbem was given, but was not able to be
found.
error in setup command: Error parsing
/home/fge/Downloads/pywbem-0.12.2/setup.cfg: Exception: Versioning for this
project requires either an sdist tarball, or access to an upstream git
repository. It's also possible that there is a mismatch between the package
name in setup.cfg and the argument given to pbr.version.VersionInfo. Project
name pywbem was given, but was not able to be found.

• Root casue pbr.version.VersionInfo failed to find out the version of current project. The pbr get version from any of below. (https://git.openstack.org/cgit/openstack-dev/pbr/tree/pbr/packaging.py)

  • Get version from git repo: _get_version_from_git().
  • Get from _get_version_from_pkg_metadata which are ['PKG-INFO', 'METADATA'].
  • Environment variable PBR_VERSION or OSLO_PACKAGE_VERSION.

• How to fix: Add PKG-INFO or METADATA file.

• Workaround: env PBR_VERSION=0.12.2 python setup.py

This commit includes OpenEnumerateInstancesWithPath, PullInstancesWithPath, CloseEnumeration, OpenEnumerateInstancePaths, and PullInstancePaths. The documentation is defined for all but OpenEnumerateInstancePaths and PullInstancePaths.

The mock test is broken but there is a first set of tests in run_cimoperations.py that executes a number of variations of OpenEnumerateInstancePaths with pull and close against OpenPegasus.

NOT READY FOR MERGE.

Here is first cut of part ready for review and comment. In particular, this currently fails mock and only about hallf of operaitons there. Since the documentaiton for each operation is enormous would like to get the first one clean (OpenEnumerateInstances) before doing the other opens.

Also some code is hacky and we should discuss the enum_context thing class.

Anyway, it actually works now.

slp is the defined discovery mechanism for SNIA smi and DMTF WBEM. However, any slp requests would have to be processed outside pywbem today and further, the only apparent python slp implementation is very old meaning that there is no easy way to introduce slp info within a python/pywbem environment. There are a couple of possibilities here:

1. a new, improved, slp interface as a github project etc.
2. let users try to use the existing repositories (see http://openslp.svn.sourceforge.net/)
3. introduce an slp interface within pwbem

Looking for ideas about what we should do

The setup-python action used in the test workflow invokes pip. On Python 2.7 and 3.5, the pip invocation displays a Python deprecation message:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as
Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. 
More details about Python 2support in pip, can be found at
https://pip.pypa.io/en/latest/development/release-process/#python-2-support

That warning shows up in the actions summary and clutters up the summary. The same happens on Python 3.5.

When running pip locally, it is possible to disable that warning by setting the environment variable:

PIP_NO_PYTHON_VERSION_WARNING=1

PR #2962 sets that environment variable in the workflow step that invokes the setup-python action. However, while the environment variable is listed in the log, it seems to be ignored because the warning is still shown.

The env var PIP_DISABLE_PIP_VERSION_CHECK=1 works as expected with the setup-python action.

Opened issue https://github.com/actions/setup-python/issues/575 to follow up.

The current implementation retries POST operations, because most POST operations in WBEM are actually read-only. Some of them are not however, and for those this can lead to duplicate execution.

See https://github.com/pywbem/pywbem/blob/master/pywbem/_cim_operations.py#L208 which sets POST as the only method to retry.

PR #2902 transitioned the pywbem distribution archive build away from using setup.py to using the "build" package. That worked for the normal wheel archive, but not for the cythonized archive. PRs #2905 and #2906 were needed to get that back to work and right now, the cythonized archive is again built using setup.py.

This issue is to transition the build of the cythonized archive build off of setuptools as a frontend.

The order of XML element names output by Open Pegasus response to class associators request CLASS NAME element differs between pywbem > 3.5 and older versions of pywbem at least in a pywbemtools test that does a class associators request against pywbem mock. The ordering of the components of the QUALIFIER NAME elements is different as shown below. In one case they appear to be lexically ordered. In the other, not so much. This appeared when a test was created to test the class associators response with --outputformat xml and using the new script simple_three-ns-model-script.py but the same class definition exists in the test file simple_mock_model

Python <= 3.5
<CLASS NAME="CIM_FooRef2" SUPERCLASS="CIM_BaseRef">
      <QUALIFIER NAME="Description" OVERRIDABLE="true" PROPAGATED="false" TOSUBCLASS="true" TRANSLATABLE="true" TYPE="string">
           <VALUE>Class 2 that is referenced</VALUE>
      </QUALIFIER>
 </CLASS>

Order output when running with python version ge 3.5

<CLASS NAME="CIM_FooRef2" SUPERCLASS="CIM_BaseRef">
    <QUALIFIER NAME="Description" OVERRIDABLE="true" PROPAGATED="false" TOSUBCLASS="true" TRANSLATABLE="true" TYPE="string">
           <VALUE>Class 2 that is referenced</VALUE>
     </QUALIFIER>
 </CLASS>

This was found when running a test

Running coverity on version 0.15.0 which is the latest packaged version for fedora is reporting some errors. I believe this PR addresses some potential issues that should be considered for inclusion.

With the upgrade of windows-latest on GitHub Actions from Windows Server 2019 to Windows Server 2022 (see here), a number of compile errors appeared in our Window environments on Python 3.5 with minimum package levels (for: cffi, psutil, pyrsistent).

By changing minimum versions of sometimes setuptools, and sometimes the package itself, the errors for cffi and psutil could be avoided. However, it is not clear what to do for pyrsistent.

While it would be possible to change the Windows level from windows-latest back to windows-2019, we wanted to do that only on the environments where it fails, and that creates additional complexity in the test workflow file.

In addition, support for Python 3.5 has created significant work in the past and should be expected to create more work in the future.

Therefore, the proposal is to drop support for Python 3.5 with pywbem 1.5.0 and pywbemtools 1.1.0.