Python 3 wrapper of Microsoft UIAutomation. Support UIAutomation for MFC, WindowsForm, WPF, Modern UI(Metro UI), Qt, IE, Firefox, Chrome ...

Overview

The uiautomation module

🇨🇳 中文版介绍

Do not use 3.7.6 and 3.8.1, comtypes doesn't work in these two versions. Install an earlier version or the latest version. https://github.com/enthought/comtypes/issues/202

This module is for UIAutomation on Windows(Windows XP with SP3, Windows Vista, Windows 7 and Windows 8/8.1/10). It supports UIAutomation for the applications which implmented UIAutomation Provider, such as MFC, Windows Form, WPF, Modern UI(Metro UI), Qt(Partly), Firefox(version<=56 or >=60), Chrome and Electron based apps(require --force-renderer-accessibility command line parameter).

I developed it in my spare time and for my personal use.

uiautomation is shared under the Apache Licence 2.0.
This means that the code can be freely copied and distributed, and costs nothing to use.

uiautomation1.x supports py2, py3 and doesn't depend on any third package.

uiautomation2.0+ only supports py3 and depends on comtypes and typing(Python3.5+ built-in).
uiautomation2.0+ is not backward compatible with early versions. See API changes.

You can install uiautomation by "pip install uiautomation". After installation, a automation.py that calls uiautomation will be in 'C:\PythonXX\Scripts'. You use this script to traverse UI controls.

Run 'C:\PythonXX\Scripts\automation.py -h' for help.
Run demos\automation_calculator.py to see a simple demo.

On Windows 8/8.1, to automate a Metro App, the app must be in foreground. If a Metro App was switched to background, uiautomation can't fetch its controls' information.

By the way, You should run python as administrator. Otherwise uiautomation may fail to enumerate controls or get controls' information on Windows 7 or higher.

Requirements:

Microsoft UIAutomation Minimum supported client: Windows 7, Windows Vista with SP2 and Platform Update for Windows Vista, Windows XP with SP3 and Platform Update for Windows Vista [desktop apps only]

Microsoft UIAutomation Minimum supported server: Windows Server 2008 R2, Windows Server 2008 with SP2 and Platform Update for Windows Server 2008, Windows Server 2003 with SP2 and Platform Update for Windows Server 2008 [desktop apps only]

C++ dll source code: UIAutomationClient


How to use uiautomation? run 'automation.py -h' help

Understand the arguments of automation.py, and try the following examples:
automation.py -t 0 -n, print current active window's controls, show fullname
automation.py -r -d 1 -t 0, print desktop(the root of control tree) and it's children(top level windows)

top level windows

automation.py prints the properties of controls and the patterns they support. You use controls and patterns to get controls' information and operate them.

A control should support some patterns or conditionally supports some patterns according to its control type.

patterns

Refer Control Pattern Mapping for UI Automation Clients for the full control pattern table.

uiautomation searches controls from the control tree based on the controls' properties you supply.

Suppose the control tree is

root(Name='Desktop', Depth=0)
  window1(Depth=1)
    control1-001(Depth=2)
    control1-...(Depth=2)
    ...
    control1-100(Depth=2)
  window2(Name='window2', Depth=1)
    control2-1(Depth=2)
      control2-1-001(Depth=3)
      control2-1-...(Depth=3)
      ...
      control2-1-100(Depth=3)
    control2-2(Depth=2)
    control2-3(Depth=2)
    control2-4(Name='2-4', Depth=2)
      editcontrol(Name='myedit1', Depth=3)
      editcontrol(Name='myedit2', Depth=3)

If you want to find the EditControl whose name is 'myedit2' and type 'hi',
you can write the following code:

uiautomation.EditControl(searchDepth=3, Name='myedit2').SendKeys('hi')

But this code run slowly because there are more than 200 controls before myedit2 in the control tree.
uiautomation has to traverse more than 200 controls before finding myedit2 if search from root in 3 search depth.
The better is:

window2 = uiautomation.WindowControl(searchDepth=1, Name='window2') # search 2 times
sub = window2.Control(searchDepth=1, Name='2-4')    # search 4 times
edit = sub.EditControl(searchDepth=1, Name='myedit2')   # search 2 times
edit.SendKeys('hi')

This code run faster than the former.
You can also combine the four lines code into one line.

uiautomation.WindowControl(searchDepth=1, Name='window2').Control(searchDepth=1, Name='2-4').EditControl(searchDepth=1, Name='myedit2').SendKeys('hi')

Now let's take notepad.exe for an example.
Luanch notepad.exe and run automation.py -t 3, then swith to Notepad and wait for 5 seconds

automation.py will print the controls of Notepad and save them to @AutomationLog.txt:

ControlType: PaneControl ClassName: #32769 Name: 桌面 Depth: 0 (Desktop window, the root control)
  ControlType: WindowControl ClassName: Notepad Depth: 1 (Top level window)
    ControlType: EditControl ClassName: Edit Depth: 2
      ControlType: ScrollBarControl ClassName: Depth: 3
        ControlType: ButtonControl ClassName: Depth: 4
        ControlType: ButtonControl ClassName: Depth: 4
      ControlType: ThumbControl ClassName: Depth: 3
    ControlType: TitleBarControl ClassName: Depth: 2
      ControlType: MenuBarControl ClassName: Depth: 3
        ControlType: MenuItemControl ClassName: Depth: 4
      ControlType: ButtonControl ClassName: Name: 最小化 Depth: 3 (Minimize Button)
      ControlType: ButtonControl ClassName: Name: 最大化 Depth: 3 (Maximize Button)
      ControlType: ButtonControl ClassName: Name: 关闭 Depth: 3 (Close Button)
...

Run the following code

# -*- coding: utf-8 -*-
import subprocess
import uiautomation as auto

print(auto.GetRootControl())
subprocess.Popen('notepad.exe')
# you should find the top level window first, then find children from the top level window
notepadWindow = auto.WindowControl(searchDepth=1, ClassName='Notepad')
if not notepadWindow.Exists(3, 1):
    print('Can not find Notepad window')
    exit(0)
print(notepadWindow)
notepadWindow.SetTopmost(True)
# find the first EditControl in notepadWindow
edit = notepadWindow.EditControl()
# use value pattern to get or set value
edit.GetValuePattern().SetValue('Hello')# or edit.GetPattern(auto.PatternId.ValuePattern)
edit.SendKeys('{Ctrl}{End}{Enter}World')
# find the first TitleBarControl in notepadWindow, 
# then find the second ButtonControl in TitleBarControl, which is the Maximize button
notepadWindow.TitleBarControl().ButtonControl(foundIndex=2).Click()
# find the first button in notepadWindow whose Name is '关闭', the close button
# the relative depth from Close button to Notepad window is 2
notepadWindow.ButtonControl(searchDepth=2, Name='关闭').Click()
# then notepad will popup a window askes you to save or not, press hotkey alt+n not to save
auto.SendKeys('{Alt}n')

auto.GetRootControl() returns the root control(the Desktop window)
auto.WindowControl(searchDepth=1, ClassName='Notepad') creates a WindowControl, the parameters specify how to search the control
the following parameters can be used
searchFromControl = None,
searchDepth = 0xFFFFFFFF,
searchInterval = SEARCH_INTERVAL,
foundIndex = 1
Name
SubName
RegexName
ClassName
AutomationId
ControlType
Depth
Compare

See Control.__init__ for the comments of the parameters.
See scripts in folder demos for more examples.

Control.Element returns the low level COM object IUIAutomationElement, Almost all methods and properties of Control are implemented via IUIAutomationElement COM API and Win32 API. when calling a control's method or property that indirectly calls Control.Element and Control.Element is None, uiautomation starts to search the control by the properties you supply. uiautomation will raise a LookupError exception if it can't find the control in uiautomation.TIME_OUT_SECOND(default 10 seconds). Control.Element will has a valid value if uiautomation finds the control successfully. You can use Control.Exists(maxSearchSeconds, searchIntervalSeconds) to check whether a control Exists, this function doesn't raise any exception. Call Control.Refind or Control.Exists to make Control.Element invalid again and uiautomation will starts a new search.

For example:

#!python3
# -*- coding:utf-8 -*-
import subprocess
import uiautomation as auto
auto.uiautomation.SetGlobalSearchTimeout(15)  # set new timeout 15


def main():
    subprocess.Popen('notepad.exe')
    window = auto.WindowControl(searchDepth=1, ClassName='Notepad')
    edit = window.EditControl()
    # when calling SendKeys, uiautomation starts to search window and edit in 15 seconds
    # because SendKeys indirectly calls Control.Element and Control.Element is None
    # if window and edit don't exist in 15 seconds, a LookupError exception will be raised
    try:
        edit.SendKeys('first notepad')
    except LookupError as ex:
        print("The first notepad doesn't exist in 15 seconds")
        return
    # the second call to SendKeys doesn't trigger a search, the previous call makes sure that Control.Element is valid
    edit.SendKeys('{Ctrl}a{Del}')
    window.GetWindowPattern().Close()  # close the first Notepad, window and edit become invalid even though their Elements have a value

    subprocess.Popen('notepad.exe')  # run second Notepad
    window.Refind()  # need to refind window, trigger a new search
    edit.Refind()  # need to refind edit, trigger a new search
    edit.SendKeys('second notepad')
    edit.SendKeys('{Ctrl}a{Del}')
    window.GetWindowPattern().Close()  # close the second Notepad, window and edit become invalid again

    subprocess.Popen('notepad.exe')  # run third Notepad
    if window.Exists(3, 1): # trigger a new search
        if edit.Exists(3):  # trigger a new search
            edit.SendKeys('third notepad')  # edit.Exists makes sure that edit.Element has a valid value now
            edit.SendKeys('{Ctrl}a{Del}')
        window.GetWindowPattern().Close()
    else:
        print("The third notepad doesn't exist in 3 seconds")


if __name__ == '__main__':
    main()

If automation.py can't print the controls you see. Maybe the controls were built by DirectUI(or CustomControl), not UI Frameworks supplied by Microsoft. In order to support UIAutomation, an UI Framework must implement UI Automation Provider.

A Microsoft UI Automation provider is a software object that exposes an element of an application's UI so that accessibility client applications can retrieve information about the element and invoke its functionality. In general, each control or other distinct element in a UI has a provider.

Microsoft includes a provider for each of the standard controls that are supplied with Microsoft Win32, Windows Forms, and Windows Presentation Foundation (WPF). This means that the standard controls are automatically exposed to UI Automation clients; you do not need to implement any accessibility interfaces for the standard controls.

If your application includes any custom controls, you need to implement UI Automation providers for those controls to make them accessible to accessibility client applications. You also need to implement providers for any third party controls that do not include a provider. You implement a provider by implementing UI Automation provider interfaces and control pattern interfaces.


Another UI tool Inspect.exe supplied by Microsoft can also be used to traverse the UI elements. It has an UI interface while my script shows UI elements in terminal. But I found that my script is more convenient sometimes.

Inspect


Some screenshots:

Batch rename pdf bookmark bookmark

Microsoft Word
Word

Wireshark 3.0 (Qt 5.12) Wireshark

GitHub Desktop (Electron App) GitHubDesktop

Pretty print dir
PrettyPrint

Owner
yin kaisheng
yin kaisheng
splinter - python test framework for web applications

splinter - python tool for testing web applications splinter is an open source tool for testing web applications using Python. It lets you automate br

Cobra Team 2.6k Dec 27, 2022
Data-Driven Tests for Python Unittest

DDT (Data-Driven Tests) allows you to multiply one test case by running it with different test data, and make it appear as multiple test cases. Instal

424 Nov 28, 2022
No longer maintained, please migrate to model_bakery

Model Mommy: Smart fixtures for better tests IMPORTANT: Model Mommy is no longer maintained and was replaced by Model Bakery. Please, consider migrati

Bernardo Fontes 917 Oct 04, 2022
automate the procedure of 403 response code bypass

403bypasser automate the procedure of 403 response code bypass Description i notice a lot of #bugbountytips describe how to bypass 403 response code s

smackerdodi2 40 Dec 16, 2022
Scalable user load testing tool written in Python

Locust Locust is an easy to use, scriptable and scalable performance testing tool. You define the behaviour of your users in regular Python code, inst

Locust.io 20.4k Jan 04, 2023
CNE-OVS-SIT - OVS System Integration Test Suite

CNE-OVS-SIT - OVS System Integration Test Suite Introduction User guide Discussion Introduction CNE-OVS-SIT is a test suite for OVS end-to-end functio

4 Jan 09, 2022
Django test runner using nose

django-nose django-nose provides all the goodness of nose in your Django tests, like: Testing just your apps by default, not all the standard ones tha

Jazzband 880 Dec 15, 2022
Declarative HTTP Testing for Python and anything else

Gabbi Release Notes Gabbi is a tool for running HTTP tests where requests and responses are represented in a declarative YAML-based form. The simplest

Chris Dent 139 Sep 21, 2022
Automated Security Testing For REST API's

Astra REST API penetration testing is complex due to continuous changes in existing APIs and newly added APIs. Astra can be used by security engineers

Flipkart Incubator 2.1k Dec 31, 2022
A utility for mocking out the Python Requests library.

Responses A utility library for mocking out the requests Python library. Note Responses requires Python 2.7 or newer, and requests = 2.0 Installing p

Sentry 3.8k Jan 03, 2023
Django-google-optimize is a Django application designed to make running server side Google Optimize A/B tests easy.

Django-google-optimize Django-google-optimize is a Django application designed to make running Google Optimize A/B tests easy. Here is a tutorial on t

Adin Hodovic 39 Oct 25, 2022
Pytest plugin for testing the idempotency of a function.

pytest-idempotent Pytest plugin for testing the idempotency of a function. Usage pip install pytest-idempotent Documentation Suppose we had the follo

Tyler Yep 3 Dec 14, 2022
XSSearch - A comprehensive reflected XSS tool built on selenium framework in python

XSSearch A Comprehensive Reflected XSS Scanner XSSearch is a comprehensive refle

Sathyaprakash Sahoo 49 Oct 18, 2022
DUCKSPLOIT - Windows Hacking FrameWork using Reverse Shell

Ducksploit Install Ducksploit Hacker setup raspberry pico Download https://githu

2 Jan 31, 2022
An AWS Pentesting tool that lets you use one-liner commands to backdoor an AWS account's resources with a rogue AWS account - or share the resources with the entire internet 😈

An AWS Pentesting tool that lets you use one-liner commands to backdoor an AWS account's resources with a rogue AWS account - or share the resources with the entire internet 😈

Brandon Galbraith 276 Mar 03, 2021
Python Moonlight (Machine Learning) Practice

PyML Python Moonlight (Machine Learning) Practice Contents Design Documentation Prerequisites Checklist Dev Setup Testing Run Prerequisites Python 3 P

Dockerian Seattle 2 Dec 25, 2022
Photostudio是一款能进行自动化检测网页存活并实时给网页拍照的工具,通过调用Fofa/Zoomeye/360qua/shodan等 Api快速准确查询资产并进行网页截图,从而实施进一步的信息筛查。

Photostudio-红队快速爬取网页快照工具 一、简介: 正如其名:这是一款能进行自动化检测,实时给网页拍照的工具 信息收集要求所收集到的信息要真实可靠。 当然,这个原则是信息收集工作的最基本的要求。为达到这样的要求,信息收集者就必须对收集到的信息反复核实,不断检验,力求把误差减少到最低限度。我

s7ck Team 41 Dec 11, 2022
Whatsapp messages bulk sender using Python Selenium.

Whatsapp Sender Whatsapp Sender automates sending of messages via Whatsapp Web. The tool allows you to send whatsapp messages in bulk. This program re

Yap Yee Qiang 3 Jan 23, 2022
An Instagram bot that can mass text users, receive and read a text, and store it somewhere with user details.

Instagram Bot 🤖 July 14, 2021 Overview 👍 A multifunctionality automated instagram bot that can mass text users, receive and read a message and store

Abhilash Datta 14 Dec 06, 2022
Testinfra test your infrastructures

Testinfra test your infrastructure Latest documentation: https://testinfra.readthedocs.io/en/latest About With Testinfra you can write unit tests in P

pytest-dev 2.1k Jan 07, 2023