Integration tests on DaprClient responses

.github/workflows/validate_examples.yaml → .github/workflows/run-tests.yaml

@@ -151,6 +151,9 @@ jobs:
           cd dapr_runtime
           ./dist/linux_amd64/release/placement --healthz-port 9091 &
           cd ..
-      - name: Check Examples
+      - name: Check examples
         run: |
           tox -e examples
+      - name: Run integration tests
+        run: |
+          tox -e integration

AGENTS.md

@@ -33,7 +33,9 @@ ext/                         # Extension packages (each is a separate PyPI packa
 └── flask_dapr/              # Flask integration  ← see ext/flask_dapr/AGENTS.md
 
 tests/                       # Unit tests (mirrors dapr/ package structure)
-examples/                    # Integration test suite  ← see examples/AGENTS.md
+├── examples/                # Output-based tests that run examples and check stdout
+├── integration/             # Programmatic SDK tests using DaprClient directly
+examples/                    # User-facing example applications  ← see examples/AGENTS.md
 docs/                        # Sphinx documentation source
 tools/                       # Build and release scripts
 ```
@@ -59,17 +61,19 @@ Each extension is a **separate PyPI package** with its own `setup.cfg`, `setup.p
 | `dapr-ext-langgraph` | `dapr.ext.langgraph` | LangGraph checkpoint persistence to Dapr state store | Moderate |
 | `dapr-ext-strands` | `dapr.ext.strands` | Strands agent session management via Dapr state store | New |
 
-## Examples (integration test suite)
+## Examples and testing
 
-The `examples/` directory serves as both user-facing documentation and the project's integration test suite. Examples are validated in CI using [mechanical-markdown](https://pypi.org/project/mechanical-markdown/), which executes bash code blocks from README files and asserts expected output.
+The `examples/` directory contains user-facing example applications. These are validated by two test suites:
 
-**See `examples/AGENTS.md`** for the full guide on example structure, validation, mechanical-markdown STEP blocks, and how to add new examples.
+- **`tests/examples/`** — Output-based tests that run examples via `dapr run` and check stdout for expected strings. Uses a `DaprRunner` helper to manage process lifecycle. See `examples/AGENTS.md`.
+- **`tests/integration/`** — Programmatic SDK tests that call `DaprClient` methods directly and assert on return values, gRPC status codes, and SDK types. More reliable than output-based tests since they don't depend on print statement formatting. See `tests/integration/AGENTS.md`.
 
 Quick reference:
 ```bash
-tox -e examples                          # Run all examples (needs Dapr runtime)
-tox -e example-component -- state_store  # Run a single example
-cd examples && ./validate.sh state_store # Run directly
+tox -e examples                                      # Run output-based example tests
+tox -e examples -- test_state_store.py               # Run a single example test
+tox -e integration                                   # Run programmatic SDK tests
+tox -e integration -- test_state_store.py            # Run a single integration test
 ```
 
 ## Python version support
@@ -107,8 +111,11 @@ tox -e ruff
 # Run type checking
 tox -e type
 
-# Validate examples (requires Dapr runtime)
+# Run output-based example tests (requires Dapr runtime)
 tox -e examples
+
+# Run programmatic integration tests (requires Dapr runtime)
+tox -e integration
 ```
 
 To run tests directly without tox:
@@ -190,9 +197,8 @@ When completing any task on this project, work through this checklist. Not every
 ### Examples (integration tests)
 
 - [ ] If you added a new user-facing feature or building block, add or update an example in `examples/`
-- [ ] Ensure the example README has `<!-- STEP -->` blocks with `expected_stdout_lines` so it is validated in CI
-- [ ] If you added a new example, register it in `tox.ini` under `[testenv:examples]`
-- [ ] If you changed output format of existing functionality, update `expected_stdout_lines` in affected example READMEs
+- [ ] Add a corresponding pytest test in `tests/examples/` (output-based) and/or `tests/integration/` (programmatic)
+- [ ] If you changed output format of existing functionality, update expected output in `tests/examples/`
 - [ ] See `examples/AGENTS.md` for full details on writing examples
 
 ### Documentation
@@ -204,7 +210,7 @@ When completing any task on this project, work through this checklist. Not every
 
 - [ ] Run `tox -e ruff` — linting must be clean
 - [ ] Run `tox -e py311` (or your Python version) — all unit tests must pass
-- [ ] If you touched examples: `tox -e example-component -- <example-name>` to validate locally
+- [ ] If you touched examples: `tox -e examples -- test_<example-name>.py` to validate locally
 - [ ] Commits must be signed off for DCO: `git commit -s`
 
 ## Important files
@@ -219,7 +225,8 @@ When completing any task on this project, work through this checklist. Not every
 | `dev-requirements.txt` | Development/test dependencies |
 | `dapr/version/__init__.py` | SDK version string |
 | `ext/*/setup.cfg` | Extension package metadata and dependencies |
-| `examples/validate.sh` | Entry point for mechanical-markdown example validation |
+| `tests/examples/` | Output-based tests that validate examples by checking stdout |
+| `tests/integration/` | Programmatic SDK tests using DaprClient directly |
 
 ## Gotchas
 
@@ -228,6 +235,6 @@ When completing any task on this project, work through this checklist. Not every
 - **Extension independence**: Each extension is a separate PyPI package. Core SDK changes should not break extensions; extension changes should not require core SDK changes unless intentional.
 - **DCO signoff**: PRs will be blocked by the DCO bot if commits lack `Signed-off-by`. Always use `git commit -s`.
 - **Ruff version pinned**: Dev requirements pin `ruff === 0.14.1`. Use this exact version to match CI.
-- **Examples are integration tests**: Changing output format (log messages, print statements) can break example validation. Always check `expected_stdout_lines` in example READMEs when modifying user-visible output.
+- **Examples are tested by output matching**: Changing output format (log messages, print statements) can break `tests/examples/`. Always check expected output there when modifying user-visible output.
 - **Background processes in examples**: Examples that start background services (servers, subscribers) must include a cleanup step to stop them, or CI will hang.
 - **Workflow is the most active area**: See `ext/dapr-ext-workflow/AGENTS.md` for workflow-specific architecture and constraints.

CLAUDE.md

@@ -1 +1,14 @@
 @AGENTS.md
+
+Use pathlib instead of os.path.
+Use httpx instead of urllib.
+subprocess(`shell=True`) is used only when it makes the code more readable. Use either shlex or args lists.
+subprocess calls should have a reasonable timeout.
+Use modern Python (3.10+) features.
+Make all code strongly typed.
+Keep conditional nesting to a minimum, and use guard clauses when possible.
+Aim for medium "visual complexity": use intermediate variables to store results of nested/complex function calls, but don't create a new variable for everything.
+Avoid comments unless there is a gotcha, a complex algorithm or anything an experienced code reviewer needs to be aware of. Focus on making better Google-style docstrings instead.
+
+The user is not always right. Be skeptical and do not blindly comply if something doesn't make sense.
+Code should be cross-platform and production ready.

README.md

@@ -121,19 +121,23 @@ tox -e py311
 tox -e type
 ```
 
-8. Run examples
+8. Run integration tests
 
 ```bash
-tox -e examples
+tox -e integration
 ```
 
-[Dapr Mechanical Markdown](https://github.com/dapr/mechanical-markdown) is used to test the examples.
+9. Validate the examples
+
+```bash
+tox -e examples
+```
 
-If you need to run the examples against a pre-released version of the runtime, you can use the following command:
+If you need to run the examples or integration tests against a pre-released version of the runtime, you can use the following command:
 - Get your daprd runtime binary from [here](https://github.com/dapr/dapr/releases) for your platform.
 - Copy the binary to your dapr home folder at $HOME/.dapr/bin/daprd.
 Or using dapr cli directly: `dapr init --runtime-version <release version>`
-- Now you can run the example with `tox -e examples`.
+- Now you can run the examples with `tox -e examples` or the integration tests with `tox -e integration`.
 
 
 ## Documentation

examples/AGENTS.md

@@ -1,14 +1,13 @@
 # AGENTS.md — Dapr Python SDK Examples
 
-The `examples/` directory serves as both **user-facing documentation** and the project's **integration test suite**. Each example is a self-contained application validated automatically in CI using [mechanical-markdown](https://pypi.org/project/mechanical-markdown/), which executes bash code blocks embedded in README files and asserts expected output.
+The `examples/` directory serves as the **user-facing documentation**. Each example is a self-contained application validated by pytest-based tests in `tests/examples/`.
 
 ## How validation works
 
-1. `examples/validate.sh` is the entry point — it `cd`s into an example directory and runs `mm.py -l README.md`
-2. `mm.py` (mechanical-markdown) parses `<!-- STEP -->` HTML comment blocks in the README
-3. Each STEP block wraps a fenced bash code block that gets executed
-4. stdout/stderr is captured and checked against `expected_stdout_lines` / `expected_stderr_lines`
-5. Validation fails if any expected output line is missing
+1. Each example has a corresponding test file in `tests/examples/` (e.g., `test_state_store.py`)
+2. Tests use a `DaprRunner` helper (defined in `tests/examples/conftest.py`) that wraps `dapr run` commands
+3. `DaprRunner.run()` executes a command and captures stdout; `DaprRunner.start()`/`stop()` manage background services
+4. Tests assert that expected output lines appear in the captured output
 
 Run examples locally (requires a running Dapr runtime via `dapr init`):
 
@@ -17,10 +16,7 @@ Run examples locally (requires a running Dapr runtime via `dapr init`):
 tox -e examples
 
 # Single example
-tox -e example-component -- state_store
-
-# Or directly
-cd examples && ./validate.sh state_store
+tox -e examples -- test_state_store.py
 ```
 
 In CI (`validate_examples.yaml`), examples run on all supported Python versions (3.10-3.14) on Ubuntu with a full Dapr runtime including Docker, Redis, and (for LLM examples) Ollama.
@@ -31,7 +27,7 @@ Each example follows this pattern:
 
 ```
 examples/<example-name>/
-├── README.md              # Documentation + mechanical-markdown STEP blocks (REQUIRED)
+├── README.md              # Documentation (REQUIRED)
 ├── *.py                   # Python application files
 ├── requirements.txt       # Dependencies (optional — many examples rely on the installed SDK)
 ├── components/            # Dapr component YAML configs (if needed)
@@ -46,53 +42,6 @@ Common Python file naming conventions:
 - Client/caller side: `*-caller.py`, `publisher.py`, `*_client.py`
 - Standalone: `state_store.py`, `crypto.py`, etc.
 
-## Mechanical-markdown STEP block format
-
-STEP blocks are HTML comments wrapping fenced bash code in the README:
-
-````markdown
-<!-- STEP
-name: Run the example
-expected_stdout_lines:
-  - '== APP == Got state: value'
-  - '== APP == State deleted'
-background: false
-sleep: 5
-timeout_seconds: 30
-output_match_mode: substring
-match_order: none
--->
-
-```bash
-dapr run --app-id myapp --resources-path ./components/ python3 example.py
-```
-
-<!-- END_STEP -->
-````
-
-### STEP block attributes
-
-| Attribute | Description |
-|-----------|-------------|
-| `name` | Descriptive name for the step |
-| `expected_stdout_lines` | List of strings that must appear in stdout |
-| `expected_stderr_lines` | List of strings that must appear in stderr |
-| `background` | `true` to run in background (for long-running services) |
-| `sleep` | Seconds to wait after starting before moving to the next step |
-| `timeout_seconds` | Max seconds before the step is killed |
-| `output_match_mode` | `substring` for partial matching (default is exact) |
-| `match_order` | `none` if output lines can appear in any order |
-
-### Tips for writing STEP blocks
-
-- Use `background: true` with `sleep:` for services that need to stay running (servers, subscribers)
-- Use `timeout_seconds:` to prevent CI hangs on broken examples
-- Use `output_match_mode: substring` when output contains timestamps or dynamic content
-- Use `match_order: none` when multiple concurrent operations produce unpredictable ordering
-- Always include a cleanup step (e.g., `dapr stop --app-id ...`) when using background processes
-- Make `expected_stdout_lines` specific enough to validate correctness, but not so brittle they break on cosmetic changes
-- Dapr prefixes app output with `== APP ==` — use this in expected lines
-
 ## Dapr component YAML format
 
 Components in `components/` directories follow the standard Dapr resource format:
@@ -182,69 +131,18 @@ The `workflow` example includes: `simple.py`, `task_chaining.py`, `fan_out_fan_i
 1. Create a directory under `examples/` with a descriptive kebab-case name
 2. Add Python source files and a `requirements.txt` referencing the needed SDK packages
 3. Add Dapr component YAMLs in a `components/` subdirectory if the example uses state, pubsub, etc.
-4. Write a `README.md` with:
-   - Introduction explaining what the example demonstrates
-   - Pre-requisites section (Dapr CLI, Python 3.10+, any special tools)
-   - Install instructions (`pip3 install dapr dapr-ext-grpc` etc.)
-   - Running instructions with `<!-- STEP -->` blocks wrapping `dapr run` commands
-   - Expected output section
-   - Cleanup step to stop background processes
-5. Register the example in `tox.ini` under `[testenv:examples]` commands:
-   ```
-   ./validate.sh your-example-name
-   ```
-6. Test locally: `cd examples && ./validate.sh your-example-name`
-
-## Common README template
-
-```markdown
-# Dapr [Building Block] Example
-
-This example demonstrates how to use the Dapr [building block] API with the Python SDK.
-
-## Pre-requisites
-
-- [Dapr CLI and initialized environment](https://docs.dapr.io/getting-started)
-- Python 3.10+
-
-## Install Dapr python-SDK
-
-\`\`\`bash
-pip3 install dapr dapr-ext-grpc
-\`\`\`
-
-## Run the example
-
-<!-- STEP
-name: Run example
-expected_stdout_lines:
-  - '== APP == Expected output here'
-timeout_seconds: 30
--->
-
-\`\`\`bash
-dapr run --app-id myapp --resources-path ./components/ python3 example.py
-\`\`\`
-
-<!-- END_STEP -->
-
-## Cleanup
-
-<!-- STEP
-name: Cleanup
--->
-
-\`\`\`bash
-dapr stop --app-id myapp
-\`\`\`
-
-<!-- END_STEP -->
-```
+4. Write a `README.md` with introduction, pre-requisites, install instructions, and running instructions
+5. Add a corresponding test in `tests/examples/test_<example_name>.py`:
+   - Use the `@pytest.mark.example_dir('<example-name>')` marker to set the working directory
+   - Use `dapr.run()` for scripts that exit on their own, `dapr.start()`/`dapr.stop()` for long-running services
+   - Assert expected output lines appear in the captured output
+6. Test locally: `tox -e examples -- test_<example_name>.py`
 
 ## Gotchas
 
-- **Output format changes break CI**: If you modify print statements or log output in SDK code, check whether any example's `expected_stdout_lines` depend on that output.
-- **Background processes must be cleaned up**: Missing cleanup steps cause CI to hang.
+- **Output format changes break tests**: If you modify print statements or log output in SDK code, check whether any test's expected lines in `tests/examples/` depend on that output.
+- **Background processes must be cleaned up**: The `DaprRunner` fixture handles cleanup on teardown, but tests should still call `dapr.stop()` to capture output.
 - **Dapr prefixes output**: Application stdout appears as `== APP == <line>` when run via `dapr run`.
 - **Redis is available in CI**: The CI environment has Redis running on `localhost:6379` — most component YAMLs use this.
 - **Some examples need special setup**: `crypto` generates keys, `configuration` seeds Redis, `conversation` needs LLM config — check individual READMEs.
+- **Infinite-loop example scripts**: Some example scripts (e.g., `invoke-caller.py`) have `while True` loops for demo purposes. Tests must either bypass these with HTTP API calls or use `dapr.run(until=...)` for early termination.

examples/distributed_lock/lock.py

@@ -27,7 +27,7 @@ def main():
         print('Will try to acquire a lock from lock store named [%s]' % store_name)
         print('The lock is for a resource named [%s]' % resource_id)
         print('The client identifier is [%s]' % client_id)
-        print('The lock will will expire in %s seconds.' % expiry_in_seconds)
+        print('The lock will expire in %s seconds.' % expiry_in_seconds)
 
         with dapr.try_lock(store_name, resource_id, client_id, expiry_in_seconds) as lock_result:
             assert lock_result.success, 'Failed to acquire the lock. Aborting.'

pyproject.toml

@@ -22,3 +22,8 @@ ignore = ["E501","E203", "E712", "E722", "E713"]
 
 [tool.ruff.format]
 quote-style = 'single'
+
+[tool.pytest.ini_options]
+markers = [
+    'example_dir(name): set the example directory for the dapr fixture',
+]