An open source development framework to help you build data workflows and modern data architecture on AWS.

Overview

AWS DataOps Development Kit (DDK)

Actions Status Actions Status Actions Status Actions Status

The AWS DataOps Development Kit is an open source development framework for customers that build data workflows and modern data architecture on AWS.

Based on the AWS CDK, it offers high-level abstractions allowing you to build pipelines that manage data flows on AWS, driven by DevOps best practices. The framework is extensible, you can add abstractions for your own data processing infrastructure or replace our best practices with your own standards. It's easy to share templates, so everyone in your organisation can concentrate on the business logic of dealing with their data, rather than boilerplate logic.


The DDK Core is a library of CDK constructs that you can use to build data workflows and modern data architecture on AWS, following our best practice. The DDK Core is modular and extensible, if our best practice doesn't work for you, then you can update and share your own version with the rest of your organisation by leveraging a private AWS Code Artifact repository.

You can compose constructs from the DDK Core into a DDK App. Your DDK App can also add contain constructs from the CDK Framework or the AWS Construct Library.

You can use the DDK CLI to manage your DDK App. You can use it to create a new app from a template, or deploy your DDK app to AWS.

Getting Started

For a detailed walk-through, check out our Workshop.

At a glance

Install or update the AWS DDK from PyPi.

pip install aws-ddk

Create a new project:

ddk init sample-app

This will create a sample-app directory inside the current folder. Inside that directory, it will generate the initial project structure, and initialize a virtual environment.

sample-app
├── .gitignore
├── .venv
├── README.md
├── app.py
├── cdk.json
├── ddk.json
├── ddk_app
│   ├── __init__.py
│   └── ddk_app_stack.py
├── requirements-dev.txt
├── requirements.txt
├── setup.py
└── source.bat

To activate the virtual environment, and install the dependencies, run:

source .venv/bin/activate && pip install -r requirements.txt

Next, let us examine the code. If you look at app.py, it will look like this:

import aws_cdk as cdk
from ddk_app.ddk_app_stack import DdkApplicationStack

app = cdk.App()
DdkApplicationStack(app, "DdkApplication", "dev")

app.synth()

If your AWS account hasn't been used to deploy DDK apps before, then you need to bootstrap your environment:

ddk bootstrap

You can then deploy your DDK app:

ddk deploy

Official Resources

Getting Help

The best way to interact with our team is through GitHub. You can open an issue and choose from one of our templates for bug reports, feature requests, or documentation issues. If you have a feature request, don't forget you can search existing issues and upvote or comment on existing issues before creating a new one.

Contributing

We welcome community contributions and pull requests. Please see CONTRIBUTING.md for details on how to set up a development environment and submit code.

Other Ways to Support

One way you can support our project is by letting others know that your organisation uses the DDK. If you would like us to include your company's name and/or logo in this README file, please raise a 'Support the DDK' issue. Note that by raising a this issue (and related pull request), you are granting AWS permission to use your company’s name (and logo) for the limited purpose described here and you are confirming that you have authority to grant such permission.

License

This project is licensed under the Apache-2.0 License.

Comments
  • (cli) Allow disabling of public access block configuration during ddk bootstrap

    (cli) Allow disabling of public access block configuration during ddk bootstrap

    Enhancement

    • (Allow disabling of public access block configuration during ddk bootstrap

    Relates

    • #112

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

    cli effort/medium p1 
    opened by malachi-constant 5
  • (bug) ddk deploy fails on windows 11 pro

    (bug) ddk deploy fails on windows 11 pro

    Describe the bug

    ddk deploy on windows fails w/o any error message or stack trace. Ordinary cdk deploy works fine.

    How to Reproduce

    Deploying DDK stacks to AWS account XXXXXXXXXXX and region us-east-1...
    ERROR - Failed to run `cdk deploy --all --require-approval never --output .ddk.out`. See output above.
    

    Expected behavior

    No response

    Your project

    No response

    Screenshots

    image

    Environment

    Provide your `pip list` output, particularly the version of the AWS DDK library you used. Providing this information may significantly improve resolution times.
    

    OS

    Windows 11 Pro

    Python version

    3.10.8

    AWS DDK version

    0.5.1

    Additional context

    No response

    bug 
    opened by kukushking 4
  • Revirew & upgrade CDK alpha packages to official

    Revirew & upgrade CDK alpha packages to official

    We still rely on a couple of alpha packages: "aws-cdk.aws-glue-alpha" = "^2.20.0a0" "aws-cdk.aws-kinesisfirehose-alpha" = "^2.20.0a0" "aws-cdk.aws-kinesisfirehose-destinations-alpha" = "^2.20.0a0"

    Looks like official kinesisfirehose package is available.

    We should review which alpha packages we can deprecate and update the code to use official CDK packages.

    P.S. Don't attach files. Please, prefer add code snippets directly in the message body.

    enhancement dependencies 
    opened by kukushking 3
  • Missing Job Glue Name at GlueTransformStage

    Missing Job Glue Name at GlueTransformStage

    Describe the bug

    When instantiating a GlueTransformStage without a predefined crawler/job the deployment of the state machine is failing.

    Missing job_name and crawler_name used by the state machine tasks.

    How to Reproduce

        glue_stage = GlueTransformStage(
            self,
            id='xpto-gluestage',
            environment_id=environment_id,
            database_name="xpto-database",
            targets=cdk_glue.CfnCrawler.TargetsProperty(
                s3_targets=[
                    cdk_glue.CfnCrawler.S3TargetProperty(
                        path=f"s3://{xpto_data.bucket_name}/raw/"
                    )
                ]
            ),
            crawler_role=glue_role,
            job_args={
                "--S3_SOURCE_PATH": xpto_data.arn_for_objects("raw/"),
                "--S3_TARGET_PATH": xpto_data.arn_for_objects("stage/"),
            }
        )
    

    Expected behavior

    No response

    Your project

    No response

    Screenshots

    image

    image

    Environment

    Provide your `pip list` output, particularly the version of the AWS DDK library you used. Providing this information may significantly improve resolution times.
    

    OS

    Linux

    Python version

    3.10

    AWS DDK version

    0.5.1

    Additional context

    No response

    bug 
    opened by jvjfranca 3
  • How to run existing Glue Job and How to create a new Glue Job

    How to run existing Glue Job and How to create a new Glue Job

    Hi Team,

    Can someone help me on how to run existing Glue Job and How to create a new Glue Job with the pyspark code files. I tried doing it multiple ways but I am unable to do it.

    An AWS case # 10860038241 created to get the help from AWS Support team but did not get the great help.

    Below is our conversation:

    Message from AWS Support on Sep 27 2022:

    It was a pleasure talking to you. Over the chime call, you explained that you were using the DDK workshop [1] and you were just trying to add a pipeline stage for the glue job with the following: code snippets.

    glue_job_stage = GlueTransformStage( self, "ddk-glue-job", environment_id=environment_id, job_name="my_glue_job", job_role=f"arn:aws:iam::512604200947:role/service-role/AWSGlueServiceRole-sample-raw-role", crawler_role=f"arn:aws:iam::512604200947:role/service-role/AWSGlueServiceRole-sample-raw-role"

        )
    

    . . . .add_stage(glue_job_stage)

    You were getting "target" property missing error but you showed me that it was an optional parameter.

    As I mentioned, I have not worked on DDK before and since this issue is not directly related to CDK, I am going to need sometime to try to set up my own DDK workshop and see if I can set up a glue pipeline.

    Please note that code development is outside of AWS Support, however, I will put forward my best effort to help you.

    Please feel free to reply if you have any other questions.

    We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

    Message from AWS Support on Sep 28, 2022:

    Hope you are doing well!

    In order to understand the issues you were facing, I went through the ddk workshop provided in the DDK GitHub[1] and set it up in my account. I did not face any issue with the workshop sample.

    Then I added my "GlueTransformStage" [2] in the ddk_app_stack.py as shown in the snippet below:

        glue_job_stage = GlueTransformStage(
            scope=self,
            id="ddk-glue-job",
            environment_id=environment_id,
    

    . . . .add_stage(glue_job_stage)

    I was getting this error "TypeError: init() missing 1 required keyword-only argument: 'executable'"

    Further, when I tried adding the job_name or job_role or both, I was getting the same error you got "AttributeError: 'NoneType' object has no attribute 'role_arn'".

    I do not understand why we are getting these errors with those parameters as they are showing optional in the doc [2] . I thought maybe there was something extra we needed to do, so I tried adding a different stage "S3EventStage" . This worked without any issues.

    Then i took a deep dive in the documentations and found this test app [3] and referring to the code, I provided the job_name and the crawler name in my DDK app and it deployed the app successfully. Please see my latest working "ddk_app_stack.py" (attached). So, it looks like there is some issue with the code because the doc says "If the Glue job or crawler names are not supplied, then they are created." And also the only required parameters are "scope", "id" and "environment_id".

    Since DDK is a community driven open source framework and not a service like CDK, we don't have a dedicated support model. You can create an issue on the GitHub [1] to get their support.

    Having said that, I am doing all I can to get a little bit more information for you on this issue. I cannot promise if I'd get an answer but if I do, by end of tomorrow, I will definitely relay it to you. Otherwise, I will be closing this case and you may continue working with them via a GitHub issue.

    Please let me know if you have any other questions.

    Have a wonderful rest of the week!

    Reference: [1] DDK GitHub https://github.com/awslabs/aws-ddk

    [2] GlueTransformStage doc https://awslabs.github.io/aws-ddk/release/latest/api/core/stubs/aws_ddk_core.stages.GlueTransformStage.html

    [3] test_basic_data_pipeline.py https://github.com/awslabs/aws-ddk/blob/main/core/tests/unit/test_basic_data_pipeline.py#L56

    We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

    Best regards, Amazon Web Services

    Attachments [ddk_app_stack.py.txt] ddk_app_stack.py.txt

    Message from AWS Support on Sep 29, 2002:

    I was able to find a little more information for you.

    In order for GlueTransformStage to create the glue job, then the executable parameter must be supplied. An example can be found in [1] .

    If you want to use the existing job, then I believe you have to provide the job_name and the crawler_name. I was not able to do it with just "job_name".

    I request you to create an issue on the DDK GitHub if you haven't already, to get their support.

    If you have any other issue related to CDK, please create a new case.

    Reference: [1] Parameter executable example https://github.com/awslabs/aws-ddk/blob/main/core/tests/unit/test_glue_transform_stage.py#L58

    [1] CDK class class JobExecutable https://docs.aws.amazon.com/cdk/api/v2/docs/@aws-cdk_aws-glue-alpha.JobExecutable.html

    We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

    Best regards, Amazon Web Services

    question 
    opened by litethesko 3
  • (core): `add_notifications` method does not send notifications as expected

    (core): `add_notifications` method does not send notifications as expected

    Describe the bug

    add_notifications method in CICDPipelineStack creates the CodeStar notification but no notification is sent when the CodePipeline pipeline fails

    How to Reproduce

    1. Create a DDK CICD pipeline and add notifications
    CICDPipelineStack(
    ...
    .add_notifications()
    
    1. An SNS topic should be created. Subscribe your email address to it
    2. Fail the pipeline. The user expects an email to be sent but it never arrives

    Expected behavior

    An email is sent when the Pipeline fails

    Your project

    No response

    Screenshots

    No response

    Environment

    Provide your `pip list` output, particularly the version of the AWS DDK library you used. Providing this information may significantly improve resolution times.
    

    OS

    Mac

    Python version

    3.7

    AWS DDK version

    0.2.1

    Additional context

    No response

    bug core p1 
    opened by jaidisido 3
  • Github Workflow: CFN Nag

    Github Workflow: CFN Nag

    Subject: Adding Github Workflow: CFN Nag

    Feature or Bugfix

    • Feature: Adding Github Workflow: CFN Nag

    Relates

    • General Workflow Setup

    • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

    opened by malachi-constant 3
  • Add Concurrency Parameter GlueTransformStage

    Add Concurrency Parameter GlueTransformStage

    Is your idea related to a problem? Please describe. When using the GlueTransformStage, the created Glue job is unable to handle concurrent glue jobs, as the concurrency is automatically set to 1. The StepFunction and Glue Job fails when I trigger multiple runs at once. There is no way to alter the concurrency of the glue job, other than creating a Glue Job independently of the GlueTransformStage.

    Describe the solution you'd like There should be a parameter that allows a user to set the Glue Job max concurrency (ie "max_concurrent_runs") when creating a Glue Job within the GlueTransformStage.

    P.S. Don't attach files. Please, prefer add code snippets directly in the message body.

    enhancement 
    opened by tommywhelan 2
  • DDK and CDK deploy not working as stated in the workshop

    DDK and CDK deploy not working as stated in the workshop

    Describe the bug

    DDK deploy is failing with:

    Deploying DDK stacks to AWS account 619876764317 and region us-east-1...
    Traceback (most recent call last):
      File "app.py", line 3, in <module>
        import aws_cdk as cdk
    ModuleNotFoundError: No module named 'aws_cdk'
    

    CDK deploy is failing with:

    Traceback (most recent call last):
      File "app.py", line 3, in <module>
        import aws_cdk as cdk
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/__init__.py", line 24837, in <module>
        from . import aws_apigateway
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/aws_apigateway/__init__.py", line 1565, in <module>
        from ..aws_certificatemanager import ICertificate as _ICertificate_c194c70b
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/aws_certificatemanager/__init__.py", line 184, in <module>
        from ..aws_cloudwatch import (
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/aws_cloudwatch/__init__.py", line 602, in <module>
        from ..aws_iam import Grant as _Grant_a7ae64f8, IGrantable as _IGrantable_71c4f5de
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/aws_iam/__init__.py", line 662, in <module>
        class AddToPrincipalPolicyResult:
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/aws_iam/__init__.py", line 667, in AddToPrincipalPolicyResult
        policy_dependable: typing.Optional[constructs.IDependable] = None,
    AttributeError: module 'constructs' has no attribute 'IDependable'
    

    How to Reproduce

    I followed the steps in the DDK workshop, and ran into the following issue when running ddk deploy:

    Deploying DDK stacks to AWS account 619876764317 and region us-east-1...
    Traceback (most recent call last):
      File "app.py", line 3, in <module>
        import aws_cdk as cdk
    ModuleNotFoundError: No module named 'aws_cdk'
    

    My assumption was that it was related to the virtual environment, and ddk running the wrong version. This proved correct when I ran which ddk, as it printed a reference to the ddk installation outside the virtual env.

    I then tried to run cdk deploy instead. However, it failed with the following error:

    Traceback (most recent call last):
      File "app.py", line 3, in <module>
        import aws_cdk as cdk
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/__init__.py", line 24837, in <module>
        from . import aws_apigateway
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/aws_apigateway/__init__.py", line 1565, in <module>
        from ..aws_certificatemanager import ICertificate as _ICertificate_c194c70b
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/aws_certificatemanager/__init__.py", line 184, in <module>
        from ..aws_cloudwatch import (
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/aws_cloudwatch/__init__.py", line 602, in <module>
        from ..aws_iam import Grant as _Grant_a7ae64f8, IGrantable as _IGrantable_71c4f5de
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/aws_iam/__init__.py", line 662, in <module>
        class AddToPrincipalPolicyResult:
      File "/Users/lutleon/Projects/ddk-demo/sample-app/.venv/lib/python3.8/site-packages/aws_cdk/aws_iam/__init__.py", line 667, in AddToPrincipalPolicyResult
        policy_dependable: typing.Optional[constructs.IDependable] = None,
    AttributeError: module 'constructs' has no attribute 'IDependable'
    

    I was able to mitigate this by running pip install aws_cdk.constructs. After I ran that command, cdk deploy was able to synthesize the resources and show the diff.

    I am on version 2.34.0 (build 633edab) of CDK. Additionally, the Python library for aws-cdk-lib is on version 2.29.0.

    Expected behavior

    No response

    Your project

    No response

    Screenshots

    No response

    Environment

    Provide your `pip list` output, particularly the version of the AWS DDK library you used. Providing this information may significantly improve resolution times.
    

    OS

    Mac

    Python version

    3.8.10

    AWS DDK version

    0.5.1

    Additional context

    No response

    bug needs-triage 
    opened by LeonLuttenberger 2
  • (cli) add 'stacks' argument for 'ddk deploy'

    (cli) add 'stacks' argument for 'ddk deploy'

    Enhancement

    • Add argument for ddk deploy to allow explicitly setting stacks to deploy

    Usage

    • ddk deploy StackA StackB

    Relates

    • #121

    By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

    opened by malachi-constant 2
  • Enable --public-access-block-configuration like in cdk bootstrap

    Enable --public-access-block-configuration like in cdk bootstrap

    Is your idea related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

    When using the --permissions-boundary and when the permissions boundary restricts s3:PutPublicAccessBlock when bootstrapping it fails.

    Describe the solution you'd like A clear and concise description of what you want to happen.

    Please add a --public-access-block-configuration like in cdk so when bootstrapping with a permissions boundary which restricts s3:PutPublicAccessBlock, the account can still be bootstrapped.

    P.S. Don't attach files. Please, prefer add code snippets directly in the message body.

    enhancement 
    opened by spssmn-aws 2
  • core: Athena_SQL Stage Update

    core: Athena_SQL Stage Update

    * handle 2 usecases
        * If query is passed to the stage then a step function is created executing only that query [current version]
        * if query is not passed, then it needs to be passed dynamically using step function event message [e.g $._query_string]
    
    enhancement core 
    opened by malachi-constant 0
  • core: RedshiftDataApi Transform Stage

    core: RedshiftDataApi Transform Stage

    * A stage that has a step function that can run a fixed or dynamic sql query against redshift (UNLOAD/COPY/SELECT) using no/low code with SDK integration of step function of redshift data-api
        * If query is passed to the stage then a step function is created executing only that query 
        * if query is not passed, then it needs to be passed dynamically using step function event message
    * *Potential Steps*
        * step1: Execute statement (query passed)
        * step2: DescribeStatment to get execution id
        * step3: Wait for execution to complete
    
    core backlog 
    opened by malachi-constant 0
  • core: Forecast Transform Stage

    core: Forecast Transform Stage

    * A stage with a step function that trains a predictor, generates forecasts, and outputs results
    * https://docs.aws.amazon.com/forecast/latest/dg/tutorial-cloudformation.html
    * This can be extended to a lot of other AWS ML models as a service (Comprehend, Personalize, Rekognition, etc.) 
    
    core backlog 
    opened by malachi-constant 0
Releases(0.6.1)
  • 0.6.1(Dec 19, 2022)

    🚀 Enhancements / Dependency Upgrades

    Core:

    • Enhancement: Support additional pipeline attributes in CICDPipelineStack construct by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/184
    • Enhancement: SqsToLambdaStage Adding support for fifo event target by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/185
    • Enhancement: SDK for pandas layer lookup utility by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/186
    • Dependencies: CDK Library Upgrade -> 2.54 by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/190

    🙏 Thanks

    We thank the following contributors/users for their work on this release:

    @kukushking, @malachi-constant

    Full Changelog: https://github.com/awslabs/aws-ddk/compare/0.6.0...0.6.1

    Source code(tar.gz)
    Source code(zip)
  • 0.6.0(Nov 1, 2022)

    🚀 Features // 💎 Enhancements // 🪲 Bug fixes

    CLI:

    • add stacks argument for 'ddk deploy' by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/129
    • bug fix: windows ddk deploy Issue by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/180

    Core:

    • support pipeline_name parameter in CICDPipeline Construct by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/161
    • bug fix: GlueTransformStage does not pass crawler name in to SFN by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/167
    • additional resource level args in stages SqsToLambdaStage & GlueTransformStage by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/170
    • support Multiple S3 Prefixes in S3 Event Stage by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/171
    • support Multiple Buckets by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/173
    • bug fix: set the required Glue crawler permissions in GlueTransformStage by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/172
    • add add_wave support to CICDPipelineStack construct by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/176
    • add max_batching_window to SqsToLambda Stage by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/179

    Full Changelog: https://github.com/awslabs/aws-ddk/compare/0.5.1...0.6.0

    Source code(tar.gz)
    Source code(zip)
  • 0.5.1(Jun 23, 2022)

    🚀 Features / Bug fixes

    CLI:

    • add --all to ddk deploy by @jaidisido in https://github.com/awslabs/aws-ddk/pull/111

    Core:

    • Glue DataBrew Stage by @b-akhil in https://github.com/awslabs/aws-ddk/pull/123
    • remove Glue log encryption by default from security config by @kukushking in https://github.com/awslabs/aws-ddk/pull/126

    🙏 Thanks

    We thank the following contributors/users for their work on this release:

    @b-akhil, @jaidisido, @kukushking, @malachi-constant, @cnfait

    Full Changelog: https://github.com/awslabs/aws-ddk/compare/0.4.0...0.5.1

    Source code(tar.gz)
    Source code(zip)
  • 0.4.0(May 16, 2022)

    🚀 Features / Bug fixes

    Core:

    • Add stage rshift operator by @kukushking in https://github.com/awslabs/aws-ddk/pull/92
    • Refactor Stage construct to include Event, StateMachine and DataStage by @jaidisido in https://github.com/awslabs/aws-ddk/pull/104
    • Dms S3<->S3 ingestion stage by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/105

    CLI:

    • Add alpha packages to cli requirements.txt template by @jaidisido in https://github.com/awslabs/aws-ddk/pull/107

    Docs 📚

    • How-to guide ddk configuration by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/99
    • Expose configuration attributes for classes in documentation by @malachi-constant in https://github.com/awslabs/aws-ddk/pull/91
    • How-To Private Code Artifact and other enhancements by @jaidisido in https://github.com/awslabs/aws-ddk/pull/102

    Full Changelog: https://github.com/awslabs/aws-ddk/compare/0.3.0...0.4.0

    🙏 Thanks

    We thank the following contributors/users for their work on this release:

    @jaidisido, @kukushking, @malachi-constant, @cnfait

    Source code(tar.gz)
    Source code(zip)
  • 0.3.0(Apr 12, 2022)

    ⚠️ Breaking changes

    • Update CDK to v2.20.0 and pin the version #83 #72
    • Replace CloudTrail with S3 EventBridge notifications in S3EventStage #83

    🚀 Features / Bug fixes

    Core:

    • Add additional_role_policy_statements to state machine stages #82
    • Expose state machine objects in stages #81
    • Add state machine alarms #76
    • Allow multiple CloudWatch alarms per stage #78
    • Add lambda layers to SQSToLambdaStage #79
    • Add security config to Glue factory #75
    • Add firehose delivery ingestion stage #71 🚀
    • Add CodeArtifact publish action #74
    • Add AppFlow stage #68 🚀
    • Add AthenaSQL stage #66 🚀
    • Add glue factory and create glue job and crawler in stage #63
    • Add data pipeline & stage alerting #55

    CLI:

    • Remove default AWS profile #61

    Docs 📚

    • Multi-account how-to #64

    🙏 Thanks

    We thank the following contributors/users for their work on this release:

    @jaidisido, @kukushking, @malachi-constant, @cnfait

    Source code(tar.gz)
    Source code(zip)
  • 0.2.0(Mar 17, 2022)

    ⚠️ Breaking changes

    • Renamed aws_ddk_core.pipelines.Stage to aws_ddk_core.pipelines.DataStage and aws_ddk_core.pipelines.Pipeline to aws_ddk_core.pipelines.DataPipeline #33

    🚀 Features / Bug fixes

    • Enable event handling in SqsToLambdaStage #54
    • Improve git commands in CLI create-repository #53
    • Add StepFunctionFactory #48
    • Use Step Functions service-sent events in GlueTransformStage #51
    • Add escape hatches for SqsToLambdaStage #47
    • Add security checks and test stages to CICD pipeline construct #28
    • Add custom CodeBuild stage to CICD pipeline construct #37
    • Add CICD pipeline notifications via SNS #43

    Docs 📚

    • Create your own DDK Data Stage How-to guide

    🙏 Thanks

    We thank the following contributors/users for their work on this release:

    @jaidisido, @kukushking, @malachi-constant, @cnfait, @vemel, @NickCorbett

    Source code(tar.gz)
    Source code(zip)
  • 0.1.0(Feb 17, 2022)

Owner
Amazon Web Services - Labs
AWS Labs
Amazon Web Services - Labs
A Telegram Userbot to play Audio and Video songs / files in Telegram Voice Chats.

VC UserBot A Telegram Userbot to play Audio and Video songs / files in Telegram Voice Chats. It's made with PyTgCalls and Pyrogram Requirements Python

조던 1 Nov 29, 2021
Retrieve information from DBLP and update BibTex files automatically

Rebib TLDR: This script retrieves information from DBLP to update your BibTex files. python rebib.py --bibfile xxx.bib It first parses the bib entries

Shangtong Zhang 49 Jan 01, 2023
Tweet stream in OBS browser source

Tweetron TweetronはOBSブラウザーソースを使用してツイートを画面上に表示するツールソフトです Windowsのみ対応 (Windows10動作確認済) ダウンロード こちらから最新版をダウンロードしてください (現在ベータテスト版を配布しています) Download ver0.0.

Cube 0 Apr 05, 2022
SpamBot.py allows you, to spam other Chat Partners etc.

SpamBot -SpamBot.py allows you, to spam other Chat Partners etc. Install If you downloaded it yet, you have to install "requirements.txt" write the di

Marco 1 Jan 16, 2022
Python lib for Embedly

embedly-python Python library for interacting with Embedly's API. To get started sign up for a key at embed.ly/signup. Install Install with Pip (recom

Embedly 80 Oct 05, 2022
Code release for "Cycle Self-Training for Domain Adaptation" (NeurIPS 2021)

CST Code release for "Cycle Self-Training for Domain Adaptation" (NeurIPS 2021) Prerequisites torch=1.7.0 torchvision qpsolvers numpy prettytable tqd

31 Jan 08, 2023
A Python library wrapping the iFixit (Dozuki) API.

A Python library wrapping the iFixit API. Status Working, but incomplete. Fully tested and documented. Hacking I highly recommend using virtualenv: [$

James Pearson Hughes 13 May 24, 2021
A bot for the [email protected] Discord server.

KittyBot - a sentient Discord bot! Key Notes An open-source, community-powered bot for the [email 

Ollie 11 Dec 06, 2022
An alternative to OpenFaaS nats-queue-worker for long-running functions

OpenFaas Job Worker OpenFaas Job Worker is a fork of project : OSCAR Worker - https://github.com/grycap/oscar-worker Thanks to Sebástian Risco @srisco

Sebastien Aucouturier 1 Jan 07, 2022
Best badge generator API to count visitors of your Repository / Account 🥇

github visitors badge A badge generator service to count visitors of your markdown file. Hello every one! In this post, I will tell you the story of m

Sᴇɴᴜ Gᴀᴍᴇʀ Bᴏʏ 〽 3 Dec 11, 2021
Mventory is an API-driven solution for Makerspaces, Tinkerers, and Hackers.

Mventory is an API-driven inventory solution for Makers, Makerspaces, Hackspaces, and just about anyone else who needs to keep track of "stuff".

Make Monmouth 107 Dec 21, 2022
Simple Discord bot for snekbox (sandboxed Python code execution), self-host or use a global instance

snakeboxed Simple Discord bot for snekbox (sandboxed Python code execution), self-host or use a global instance

0 Jun 25, 2022
Coinbase Pro API interface framework and tooling

neutrino This project has just begun. Rudimentary API documentation Installation Prerequisites: Python 3.8+ and Git 2.33+ Navigate into a directory of

Joshua Chen 1 Dec 26, 2021
A basic Ubisoft API wrapper created in python.

UbisoftAPI A basic Ubisoft API wrapper created in python. I will be updating this with more endpoints as time goes on. Please note that this is my fir

Ethan 2 Oct 31, 2021
Wetterdienst - Open weather data for humans

We are a group of like-minded people trying to make access to weather data in Python feel like a warm summer breeze, similar to other projects like rdwd for the R language, which originally drew our

226 Jan 04, 2023
Cleiton Leonel 4 Apr 22, 2022
A working bypass for discord gc spamming

IllusionGcSpammer A working bypass for discord gc spamming Installation Run pip install pip install DiscordGcSpammer then your good to go. Usage You c

6 Sep 30, 2022
Anti-league-discordbot - Harrasses imbeciles for playing league of legends

anti-league-discordbot harrasses imbeciles for playing league of legends Running

Chris Clem 2 Feb 12, 2022
BioThings API framework - Making high-performance API for biological annotation data

BioThings SDK Quick Summary BioThings SDK provides a Python-based toolkit to build high-performance data APIs (or web services) from a single data sou

BioThings 39 Jan 04, 2023
Python library for the DeepL language translation API.

The DeepL API is a language translation API that allows other computer programs to send texts and documents to DeepL's servers and receive high-quality translations. This opens a whole universe of op

DeepL 535 Jan 04, 2023