A Python library that helps data scientists to infer causation rather than observing correlation.

Description

when i use 'from_pandas' to learning causal map by notears,i run 'watch -n 1 free -m',it shows that 3/16GB used.i run 370 thousand data but only use memory 3G?how to improve efficiency？

Context

Every 1.0s: free -m Tue Jun 30 16:18:36 2020

          total        used        free      shared  buff/cache   available

Mem: 16384 2799 12213 0 1371 13584 Swap: 0 0 0

Description

Change your installation docs to have an example of installing a version of Python that is compatible with this module.

Context

If the software is made for Python 3.5, 3.6, or 3.7 then the installation examples shouldn't show someone how to install Python 3.8

Possible Implementation

Change the 3.8 to 3.5,6, or 7.

Possible Alternatives

(Optional) Describe any alternative solutions or features you've considered.

Hi, After learn the BN model, I use the 'sm.egdes(data)' and then I find some negative weight edges in my model. What's the meaning of those negative weight edges?

Description

I want to know an input data and result for dynotears.

Context

I tried to use dynotears.from_pandas using DREAM4 challenge data, but get an empty graph. I constructed a list of dataframe as below that contains 10 dataframes. For each dataframe, the column is node and the row is timepoint such as below. g1 g2 1 1 2
2 4 2 3 3 1

Description

Your documentation lists a demo that is using the load_boston dataset to explain how to use the tool here. It also lists the variables in the dataset and you can confirm the contents.

- B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town

One of the variables used to predict a house price is skin color. That's incredibly problematic.

Given that this project is backed by a global consultancy firm I'm especially worried. People will look at this example and copy it. Since documentation pages are typically seen as an authoritative source of truth it's really dubious to use this dataset without even a mention of the controversy in the variables. The use of skin color to predict a house price is legitimately a bad practice and the documentation currently makes no effort to acknowledge it. For a package that's all about causal inference, you'd expect it to acknowledge that you still need to understand the dataset that you feed in.

Note that this dataset is up for removal from scikit-learn because of the obvious controversy and it's also something that's been pointed out at many conferences. Here is one talk from me if you're interested.

Motivation and Context

Why was this PR created?

How has this been tested?

What testing strategies have you used?

Checklist

• [ ] Read the contributing guidelines
• [ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• [ ] Updated the documentation to reflect the code changes
• [ ] Added a description of this change and added my name to the list of supporting contributions in the RELEASE.md file
• [ ] Added tests to cover my changes
• [ ] Assigned myself to the PR

Notice

• [ ] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

• I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.

• I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorised to submit this contribution on behalf of the original creator(s) or their licensees.

• I certify that the use of this contribution as authorised by the Apache 2.0 license does not violate the intellectual property rights of anyone else.

Hi, I dont know the meaning of the parameter 'w_threshold' in "from_pandas",because I can get a BN model when I using ''hill_climb" by pgmpy.The edges' number of the model vary the value of w_threshold ,so I dont know which one is correct? this problem is not exist in ''hill_climb".

Hi QB––

Description

I am running a do-calculus on a small dataset (116x32) with 2 to 4 discretized buckets. The BN fits the CPDs in 2 sec, so relatively good perf.

However a simple do-intervention takes forever and even never ends running, I waited several hours then I interrupted kernel.

Steps to Reproduce

$ from causalnex.inference import InferenceEngine $ ie = InferenceEngine(bn) $ ie.do_intervention("cD_TropCycl", {1: 0.2, 2: 0.8}) $ print("distribution after do", ie.query()["cD_TropCycl"])

Expected Result

Shouldn't it be running just a few seconds given the low number of buckets? How long does it normally take?

Actual Result

no results returned after hours running a simple query.

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• CausalNex version used (pip show causalnex): 0.5.0
• Python version used (python -V): python 3.7.6
• Operating system and version: osx 10.15.14 on 2.3 GHz Quad-Core Intel Core i5

Thank you very much!!

Motivation and Context

Quick bug fix: During the optimizer step the DAG parameters are being replaced with tensors that live on the CPU rather than on the GPU. This change basically make sure these tensors are being sent to the same device the model was originally on.

Without this fix, you will get error: "Expected all tensors to be on the same device..."

Longer term solution: I would refactor the device assignment - rather than relying on default dtype or calling to(device), create a single call at model creation that sets the dtype and device and use that for all layers and parameters.
I used Black Linter. Will add test and ensure to use the same linter as the project after my paper deadline, this quick fix to help anyone else who was stuck on this :)

How has this been tested?

Ran few examples on my data.

Checklist

• [x] Read the contributing guidelines
• [x] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• [ ] Updated the documentation to reflect the code changes
• [ ] Added a description of this change and added my name to the list of supporting contributions in the RELEASE.md file
• [ ] Added tests to cover my changes
• [x] Assigned myself to the PR

Notice

• [x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

• I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.

• I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorised to submit this contribution on behalf of the original creator(s) or their licensees.

• I certify that the use of this contribution as authorised by the Apache 2.0 license does not violate the intellectual property rights of anyone else.

Notice

• [x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

• I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.

• I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorised to submit this contribution on behalf of the original creator(s) or their licensees.

• I certify that the use of this contribution as authorised by the Apache 2.0 license does not violate the intellectual property rights of anyone else.

Motivation and Context

Why was this PR created?

• [x] Fix optional dependencies: .structure imported pytorch
• [x] Make optional pygraphviz dependency clearer: raise warning and add to README
• [ ] Test on windows machine if we catch the right errors

How has this been tested?

What testing strategies have you used?

Checklist

• [x] Read the contributing guidelines
• [x] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• [x] Updated the documentation to reflect the code changes
• [ ] Added a description of this change and added my name to the list of supporting contributions in the RELEASE.md file
• [ ] Added tests to cover my changes
• [x] Assigned myself to the PR

tryng to run the model on google collab and pandas rises issues about importing some packages the first one that turned out to be fatal is relevent to 'OrderedDict' code: /usr/local/lib/python3.6/dist-packages/pandas/io/formats/html.py in () 8 from textwrap import dedent 9 ---> 10 from pandas.compat import OrderedDict, lzip, map, range, u, unichr, zip 11 12 from pandas.core.dtypes.generic import ABCMultiIndex

ImportError: cannot import name 'OrderedDict'

the second one which is fatal is relevant to importing lmap

/usr/local/lib/python3.6/dist-packages/pandas/core/config.py in () 55 56 import pandas.compat as compat ---> 57 from pandas.compat import lmap, map, u 58 59 DeprecatedOption = namedtuple('DeprecatedOption', 'key msg rkey removal_ver')

ImportError: cannot import name 'lmap'

with standalone pandas, I did not get these issues seems to me google collab missing something any suggestions please ?

This PR was automatically created by Snyk using the credentials of a real user.

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • test_requirements.txt

mdlp-discretization 0.3.3 requires scikit-learn, which is not installed.
mdlp-discretization 0.3.3 requires numpy, which is not installed.
mdlp-discretization 0.3.3 requires scipy, which is not installed.
matplotlib 3.5.3 requires numpy, which is not installed.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 551/1000
Why? Recently disclosed, Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-SETUPTOOLS-3180412 | setuptools:
39.0.1 -> 65.5.1
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)

This PR was automatically created by Snyk using the credentials of a real user.

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • doc_requirements.txt

Sphinx 3.5.4 has requirement docutils<0.17,>=0.12, but you have docutils 0.19.
sphinx-rtd-theme 0.5.2 has requirement docutils<0.17, but you have docutils 0.19.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 551/1000
Why? Recently disclosed, Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-SETUPTOOLS-3180412 | setuptools:
39.0.1 -> 65.5.1
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)

This PR was automatically created by Snyk using the credentials of a real user.

Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

Changes included in this PR

• Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
  • requirements.txt

statsmodels 0.0.0 requires pandas, which is not installed.
statsmodels 0.0.0 requires numpy, which is not installed.
scipy 1.6.3 requires numpy, which is not installed.
scikit-learn 0.24.2 requires numpy, which is not installed.
pgmpy 0.1.19 requires pandas, which is not installed.
pgmpy 0.1.19 requires numpy, which is not installed.
patsy 0.5.3 requires numpy, which is not installed.

Vulnerabilities that will be fixed

By pinning:

Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- | 551/1000
Why? Recently disclosed, Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
SNYK-PYTHON-SETUPTOOLS-3180412 | setuptools:
39.0.1 -> 65.5.1
| No | No Known Exploit

(*) Note that the real score may have changed since the PR was raised.

Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

For more information: 🧐 View latest project report

🛠 Adjust project settings

📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:

🦉 Regular Expression Denial of Service (ReDoS)

Description

As discovered by many during a recent hackathon workshop in SG, we found that many users, (mostly Windows & Mac) faced problems installing through pip / conda.

Furthermore, due to a recent update in pygraphviz, it broke tutorials at that time.

Context

I do find this library's direction good, however, could use more stability, especially when workshops are conducted with it.

Possible Implementation

GitHub Actions (Free) can run pip install / conda install pipelines on many environments. See: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

The workflow may be something like this:

| Envs | Windows | Mac | Ubuntu | |---------------------|---------|--------|--------| | Installation Method | | | | | pip | Job 1 | Job 2 | ... | | pip w/ pygraphviz | | | | | conda | | | | | conda w/ pygraphviz | | | Job n |

Note that dimensionality can be reduced through matrices. Which takes a cartesian product of lists provided. See: https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs

Description

I was trying to determine a single latent variable in my model, and when I tried to run the EM algorithm using fit_latent_cpds, it sometimes throw random errors while some times it can product some result.

Steps to Reproduce

I have created the following test data to try the model:

data = pd.DataFrame({'node1': np.repeat(1, 50), 'node2': np.repeat(1,50)})
for i in [0, 3, 5, 13, 17, 29, 30, 31, 32]:
    data['node1'][i] = 0

for i in [4,5,11,15,17,25,27,34,41,47]:
    data['node2'][i] = 0

The data structure is very simple, a latent variable latent1 that affects node1 and node2.

sm = StructureModel()
sm.add_edges_from([('latent1', 'node1'), ('latent1', 'node2')])
bn = BayesianNetwork(sm)
bn.node_states = {'latent1':{0,1}, 'node1': {0,1}, 'node2': {0,1}}
bn.fit_latent_cpds(lv_name="latent1", lv_states=[0, 1], data=data[["node1", "node2"]], n_runs=30)

Some times I received good result as following:

{'latent1':                  
latent1          
0        0.283705
1        0.716295,

'node1': latent1        0         1
node1                     
0        0.21017  0.168051
1        0.78983  0.831949,

'node2': latent1         0         1
node2                      
0        0.253754  0.178709
1        0.746246  0.821291}

However, some times I receive different error messages:

Traceback (most recent call last):
  File "test_2.py", line 28, in <module>
    bn.fit_latent_cpds(lv_name="latent1", lv_states=[0, 1], data=data[["node1", "node2"]], n_runs=30)
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/causalnex/network/network.py", line 553, in fit_latent_cpds
    estimator = EMSingleLatentVariable(
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/causalnex/estimator/em.py", line 144, in __init__
    self._mb_data, self._mb_partitions = self._get_markov_blanket_data(data)
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/causalnex/estimator/em.py", line 585, in _get_markov_blanket_data
    mb_product = cpd_multiplication([self.cpds[node] for node in self.valid_nodes])
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/causalnex/utils/pgmpy_utils.py", line 122, in cpd_multiplication
    product_pgmpy = factor_product(*cpds_pgmpy)  # type: TabularCPD
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pgmpy/factors/base.py", line 76, in factor_product
    return reduce(lambda phi1, phi2: phi1 * phi2, args)
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pgmpy/factors/base.py", line 76, in <lambda>
    return reduce(lambda phi1, phi2: phi1 * phi2, args)
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pgmpy/factors/discrete/DiscreteFactor.py", line 930, in __mul__
    return self.product(other, inplace=False)
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pgmpy/factors/discrete/DiscreteFactor.py", line 697, in product
    phi = self if inplace else self.copy()
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pgmpy/factors/discrete/CPD.py", line 299, in copy
    return TabularCPD(
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pgmpy/factors/discrete/CPD.py", line 142, in __init__
    super(TabularCPD, self).__init__(
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/pgmpy/factors/discrete/DiscreteFactor.py", line 99, in __init__
    raise ValueError("Variable names cannot be same")
ValueError: Variable names cannot be same

And sometimes I receive this error:

Traceback (most recent call last):
  File "test_2.py", line 28, in <module>
    bn.fit_latent_cpds(lv_name="latent1", lv_states=[0, 1], data=data[["node1", "node2"]], n_runs=30)
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/causalnex/network/network.py", line 563, in fit_latent_cpds
    estimator.run(n_runs=n_runs, stopping_delta=stopping_delta)
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/causalnex/estimator/em.py", line 181, in run
    self.e_step()  # Expectation step
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/causalnex/estimator/em.py", line 233, in e_step
    results = self._update_sufficient_stats(node_mb_data["_lookup_"])
  File "/Users/user/opt/anaconda3/envs/py38/lib/python3.8/site-packages/causalnex/estimator/em.py", line 448, in _update_sufficient_stats
    prob_lv_given_mb = self._mb_product[mb_cols]
KeyError: (nan, 0.0)

My code originally also includes the boundaries and priors, however I realise these two errors just randomly pop up at different times.

Please let me know if I have done something wrong in setting up the network.

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• CausalNex version used (pip show causalnex): 0.11.0
• Python version used (python -V): 3.8.15 (via conda)
• Operating system and version: Mac OS M1

Description

Trying to install causalnex on new M1 chip using poetry generates the following issue:

Because causalnex (0.11.0) depends on scikit-learn (>=0.22.0,<0.22.2.post1 || >0.22.2.post1,<0.24.1 || >0.24.1,<0.25.0)
 and no versions of causalnex match >0.11.0,<0.12.0, causalnex (>=0.11.0,<0.12.0) requires scikit-learn (>=0.22.0,<0.22.2.post1 || >0.22.2.post1,<0.24.1 || >0.24.1,<0.25.0).
So, because your package depends on both scikit-learn (^1.1.3) and causalnex (^0.11.0), version solving failed.

Unfortunately downgrading the version of scikit-learn generates a numpy compatibility error.

  ImportError: numpy is not installed.
        scikit-learn requires numpy >= 1.11.0.
        Installation instructions are available on the scikit-learn website: http://scikit-learn.org/stable/install.html

Context

I migrated to a new computer that has the Apple M1 chip

Steps to Reproduce

1. Setting up python environment (in my case: 3.8.15)
2. Installing poetry globally (installation guide)
3. Run poetry add causalnex to resolve package dependency and install causalnex

Expected Result

I was expecting causalnex to be compatible with at least scikit-learn 1.1.3

Actual Result

I can't install causalnex.

-- If you received an error, place it here.

poetry add causalnex

Because causalnex (0.11.0) depends on scikit-learn (>=0.22.0,<0.22.2.post1 || >0.22.2.post1,<0.24.1 || >0.24.1,<0.25.0)
 and no versions of causalnex match >0.11.0,<0.12.0, causalnex (>=0.11.0,<0.12.0) requires scikit-learn (>=0.22.0,<0.22.2.post1 || >0.22.2.post1,<0.24.1 || >0.24.1,<0.25.0).
So, because your package depends on both scikit-learn (^1.1.3) and causalnex (^0.11.0), version solving failed.

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• CausalNex version used (pip show causalnex):
• Python version used (python -V): 3.8.15
• Operating system and version: Apple M1