rounakbende/rh-swe-bench
rh-swe-bench: Harbor dataset on Hugging Face with 337 tasks. A Red Hat-specific benchmark for evaluating coding agents, packaged in Harbor format. Contains 337 tasks across 24 Red Hat ecosystem repositories. The source code and task packages are maintained at redhat-et/rh-swe-bench. The candidate…
Tasks
- YAML parsing for playbook verification loses mapping contents and mishandles documents containing simple keys. Valid mappings should…
- Playbook verification can fail on valid YAML containing inline lists. Loading and writing supported YAML should preserve its values and…
- Netstat statistics containing both a colon and a numeric count, such as "destination unreachable: 107", produce incorrect statistic names…
- Boot-entry inspection can report unresolved GRUB environment references or kernel arguments inconsistent with the configured command line…
- Fetching egg/core updates fails under current upload configurations, including forced refreshes and requests with cached ETags. Updates…
- Inspecting resolver configuration can produce incorrect values for repeated directives or missing fields for sparse files. Preserve…
- Certificate-expiration reporting can be skipped despite usable certificate information being available. Supported certificate sources…
- Reported XFS quota information can be incomplete or disagree with the command output, including reports covering several devices. Preserve…
- Ansible Galaxy collection installation and listing can fail or return inconsistent information. This affects repeated or equivalent…
- Error output from remote PowerShell commands can be truncated or corrupted when CLIXML is mixed with ordinary text or spans multiple…
- Collection installation can fail for supported Galaxy requirements, while malformed or incomplete entries can produce confusing outcomes…
- Templating access notifications are silently dropped after a recent change. Context-aware value tracking stops receiving notifications…
- Ansible template and plugin failures can produce diagnostic output that omits the original cause and surrounding context. Preserve the…
- Jinja2 filter and test plugins referenced by their fully qualified collection names fail to load, leaving otherwise valid templates…
- Running a Molecule scenario again with configured inventory variables can fail even when the configuration has not changed. Configured and…
- Molecule logs look different in captured output and terminal sessions. Repeat runs with inventory variables and check operations can also…
- Molecule scenario commands can fail unexpectedly or omit expected action results, and their terminal and captured logs no longer have the…
- Molecule's destroy command crashes when --destroy=never is passed or when no provisioner is configured. It should skip destruction…
- Selecting a platform that is unavailable in a Molecule scenario can let execution continue without a useful failure. Other scenarios can…
- Molecule's captured output can be inconsistent after a flush, and completed scenario runs may leave temporary state behind when their…
- Repeated Molecule create commands can attempt unnecessary work or fail even when instances already exist. Creation should remain…
- Molecule runs do not consistently follow the configured scenario sequence, with or without shared infrastructure. Failures can leave…
- With a default Molecule scenario, commands can run against that scenario even when they are absent from its configured sequence. Results…
- Molecule's generated commands can mishandle pass-through arguments and boolean options, including an explicit request to disable a…
- Reading Molecule platform information can unexpectedly alter the supplied configuration and affect later output. Platform definitions…
- Pruning a Molecule scenario without a configured provisioner raises AttributeError. Pruning should handle this configuration and preserve…
- Molecule no longer warns when molecule.yml has changed since the scenario was created. Users can continue running a scenario whose…
- Tools invoked by Molecule can reject commands containing configured short options. Equivalent option mappings can also produce different…
- Running several Molecule scenarios with shared infrastructure can repeat or omit lifecycle operations and leave resources behind after a…
- Molecule string-valued enum classes regress on non-string input: constructing an enum member from a value such as an integer now raises…
- Molecule fails to find bundled driver playbooks in the provisioner's configured location. Restore support for those configured playbooks…
- Molecule commands can fail to locate scenario files or interpret relative paths inconsistently when a scenario configuration is supplied…
- Cleanup of a delegated-driver scenario unexpectedly leaves disposable state behind even when no safe files were configured. Honor the…
- Molecule runs execute playbook tasks meant to be excluded and omit expected change details in debug output. Configured executor arguments…
- Molecule commands can exit without an explanation or report failure after a successful operation. Exit codes should reflect the actual…
- Running Molecule from a subdirectory can use the wrong project configuration when incidental repository-like metadata exists inside the…
- Molecule misses valid nested scenarios in Ansible collections or selects the wrong scenarios when names share a basename…
- Molecule's exclude patterns stop working when you pass glob-style wildcards like appliance to --exclude . It also breaks scenario…
- Failed Molecule commands can leave callers and CLI users without the supplied explanation. Invalid subcommand requests should produce…
- Molecule has started printing end-of-run reports and command borders even when users have not enabled them. These experimental features…
- Copier put operations produce incorrect ownership and permissions when different settings are requested for files and directories. Each…
- Container hooks can run without their supplied arguments, and incomplete or conflicting hook definitions can produce confusing outcomes…
- The containers/image blobcache transport does not retain expected image-layer data during pushes, so repeated pushes upload the same…
- Rootless containers can fail to start or receive unexpected file ownership with supported UID/GID mappings, including parent-namespace…
- A Quadlet label value containing spaces can be shortened unexpectedly when it is not quoted, leaving dependent services misconfigured even…
- The graph-storage template-creation test no longer reliably checks the existing driver contract. Repair the test scenario and its…
- etcd cluster membership changes are unreliable. Newly added members can be absent from subsequent membership queries, learner promotion…
- etcd storage initialization can fail despite a successful database open, and configured database options and logging are not consistently…
- Etcd startup can report inconsistent member versions, and cluster integrity checks can give misleading warnings or fail to recognize…
- Etcd startup and upgrades can report inconsistent peer versions or misleading integrity-check results. Member removal can fail for…
- etcd benchmark reports show inaccurate latency statistics and implausibly high requests per second. Reported average, standard deviation…
- etcd cluster management reports incomplete member-version information and can negotiate an incorrect cluster version during upgrades. Hash…
- When etcd's bounded v2store event history reaches capacity, subsequent operations can return stale events or lose the expected event…
- Equivalent endpoint addresses can be treated differently in etcd network operations, and Linux interface queries can return information…
- etcd benchmark reports show inaccurate averages, standard deviations, elapsed times, and requests per second, including in weighted…
- etcd member removal can reject authorized, quorum-safe requests and allow requests that should be rejected. Voting-member queries can also…
- Selecting feature-gate settings can change features outside the requested scope. Equivalent endpoint addresses can also produce…
- Queries over stored intervals can miss matching entries or return unexpected ranges for string and integer keys. Separately, command-line…
- etcd learner promotion can panic for an unknown member and leave learner metrics inconsistent after promotion. Learner-limit validation…
- During etcd upgrades, member-version reports can be incomplete and the selected cluster version can disagree with the participating…
- Etcd can fail to start with valid database settings, panic during membership operations, or reject a supported recovery workflow. Recovery…
- Etcd endpoint operations can behave inconsistently for equivalent supported address forms. On Linux, a query for an interface address may…
- Queries over stored intervals can miss overlaps or return unexpected ranges. Other affected operations include feature settings changing…
- Etcd feature settings can affect features outside the requested scope. URL-based operations also treat some equivalent endpoints…
- Etcd member-removal requests can be rejected unexpectedly or proceed when permissions or cluster availability prohibit them. During…
- etcd load-test reports contain inaccurate latency averages, standard deviations, elapsed times, and throughput, including weighted…
- Registering another watcher for an etcd v2store key can stop earlier watchers from receiving events. All active watchers should continue…
- etcd loses or misorders events once v2store history reaches capacity. Member registration during discovery does not follow the expected…
- Adding a watcher to an etcd v2store key can stop existing watchers from receiving updates. Active watchers should all receive relevant…
- etcd member registration during discovery can continue beyond the retry limit. No-space alarms do not reliably restrict writes, member…
- etcd member removal does not consistently enforce permissions and quorum safety. Restored NOSPACE alarms also fail to restrict writes…
- The supported etcd restore procedure for a single-member cluster can no longer add a new voting member. Restore that recovery workflow…
- etcd interval-tree queries return incorrect or missing results after deletions, and string-key ordering is unreliable. Preserve the…
- Queries over stored intervals can miss entries or return unexpected matches after deletions. Results should continue to reflect the…
- Comma-separated URL flags for etcd accept malformed endpoint values instead of rejecting them. Validate each supplied endpoint and return…
- Data written through the page-aligned writer can be lost or reordered when writes span page boundaries or arrive in several parts…
- etcd's auth checks are too strict for read-only cluster and maintenance operations. Listing members or getting alarm status requires full…
- With etcd authentication enabled, a transaction can attach a lease that the caller is not permitted to use. Apply the same authorization…
- Updating an etcd member's peer URLs appears to succeed, but subsequent membership queries still report the previous URLs. Ensure…
- API resource discovery can return missing or incorrect resource information from supported protobuf responses. Resource names, kinds…
- JSON output for supported resource objects can differ depending on whether streaming is used. Preserve list metadata and items and any…
- Converting an API object between supported versions can choose an unexpected kind when several candidate kinds are available. Preserve the…
- Quantity.String() can return a truncated, empty, or otherwise incorrect representation on its first use. Resource quantities such as…
- Resource lookup can miss supported singular or plural names or return an unexpected resource. Preserve established matching precedence and…
- kubectl taint mishandles node names, --all , argument validation, and dry-run requests. Tainting a named node should affect the requested…
- kubectl logs can mishandle pod selection, empty results, and output streams. Interactive exec sessions may use unexpected stdin or…
- kubectl exec mypod -- ls no longer behaves consistently with the equivalent resource/name form. Restore supported pod-name and…
- kubectl apply produces incorrect ApplySet tracking metadata and loses or unexpectedly adds last-applied-configuration annotations…
- kubectl get --sort-by returns resources in an unexpected order for supported numeric and text fields. File-based custom-column output also…
- kubectl debug accepts unsupported flag combinations and rejects valid image or argument settings depending on whether --copy-to is used…
- A Deployment created with a requested container port can omit that port. Service creation can also reject supported headless or other…
- Some kubectl apply requests with incompatible options proceed unexpectedly or produce misleading feedback. kubectl config view may show a…
- Node drain fails to recognize eviction support on some supported clusters and can continue waiting after cancellation or the configured…
- Containers created by kubectl debug with the restricted profile can lack expected protections. Separately, multiline guidance in kubectl…
- A failed cordon/drain request can leave the caller's node state showing a change that was never accepted by the server. Restore accurate…
- kubectl create secret can reject supported source inputs or produce incomplete data. Token creation can also report an outcome…
- Resource deletion can report misleading results, omit normal output, or finish at an unexpected time. Honor the requested grace period…
- kubectl get rejects valid custom-column specifications and mishandles invalid ones. YAML and other requested output formats can produce…
- ClusterRoleBindings created with service-account subjects can omit the requested subjects. Token creation also mishandles some supported…
- Applying configurations with null-valued fields can produce unexpected resource changes, and supported OpenAPI v3-backed requests can…
- kubectl delete can affect an unexpected resource selection or disregard the requested deletion timing and missing-resource behavior. Its…
- kubectl get --sort-by can return an incomplete or only partially sorted collection for large result sets. Return the full collection in…
- Kubectl creation commands can fail with supported ClusterRole aggregation settings, produce CronJobs in an unexpected namespace, or…
- Copied debug pods can differ unexpectedly from the requested metadata, container settings, or profiles. Image and selector updates can…
- Exposed services can differ from their requested selectors, ports, and service settings. Apply invocations can produce misleading feedback…
- Copies involving directories or symlinks can produce incomplete contents or unexpected destination paths. Pod log requests can also…
- Kubectl container descriptions can omit configured security information or misrepresent ports and devices. Metrics totals can appear in an…
- Apply operations can unexpectedly lose or alter last-applied configuration metadata. set-last-applied can also reject supported updates or…
- Nested-field requests to kubectl explain can fail or display unreadable descriptions. DaemonSet rollout-status requests and apply metadata…
- kubectl apply can fail on supported requests or appear successful when the requested operation did not complete. Dry-run output and forced…
- Apply requests can accept incompatible options or reject supported combinations. Some requests using OpenAPI v3 schemas also fail…
- Repeated taint updates can fail unexpectedly or report the wrong operation, including overwrite and dry-run requests. Retried file copies…
- Environment updates can leave resources unchanged for supported input sources. Updating resource limits for selected containers can also…
- Pod metrics requests can report misleading success when metrics are unavailable or behave inconsistently when no pods match. Waiting for a…
- kubectl cp can reject supported local-to-pod copies. Interrupted copies using retries may resume with missing or duplicated data, and some…
- Copied debug pods can differ from the requested metadata, container settings, or profiles. kubectl get can also omit entries from sorted…
- Copied debug pods can lose requested metadata or container settings and fail to reflect the selected profiles. Local selector updates can…
- kubectl debug can produce an incorrectly configured ephemeral container when a profile is applied, especially when the pod already has…
- Supported apply requests can fail before completing, while some invalid requests appear to succeed without doing the requested work…
- Apply requests and config views can behave inconsistently with the supplied options or selected context. ClusterRole aggregation, CronJob…
- Apply requests with incompatible options can proceed or produce misleading feedback. Restricted debug containers can also lack expected…
- Exposed services can differ from the requested settings or be created from unsupported requests. Sorted resource listings can also omit…
- Nested-field requests to kubectl explain can fail or produce unreadable descriptions. Markdown-based command help, DaemonSet rollout…
- Exposing a resource can create a service with unexpected selectors, ports, or service settings. Restricted debug sessions can lack…
- Copied debug pods can receive unexpected metadata or fail to reflect requested container settings and profiles. Resource listings can also…
- kubectl cp can reject valid pod/file arguments or use an unexpected destination path. Supported local and remote copy forms should work…
- Taint updates can report the wrong operation or mishandle overwrite and dry-run requests. Retried file copies can lose continuity, and pod…
- Exec requests can fail for supported pod/command forms or use unexpected stdin and terminal settings. Port-forward requests can also be…
- Creating jobs and services can fail with supported inputs, while incomplete or incompatible requests can be accepted. Honor supported job…
- kubectl create mishandles editing, extra arguments, dry-run requests, and annotation failures. Restore editing of the proposed resource…
- Creating ConfigMaps and Secrets with kubectl can produce missing data or unexpected failures from supported inputs. Created objects should…
- Apply requests can produce unexpected outcomes for incompatible options, and config views may show a different context from the selected…
- Some supported pod copy requests fail or produce incomplete contents and unexpected directory paths. Pod log requests can also mishandle…
- Pod log requests can produce unexpected selection, empty-result, or streaming behavior. Interactive exec and port-forward requests can…
- kubectl debug can return the wrong ephemeral container or a result that lacks the requested custom-profile settings. Restore profile…
- Taint requests can be accepted or rejected unexpectedly when selecting nodes by name or selector. Removal requests may also appear…
- ApplySet operations mishandle custom-resource parent validation and object-listing failures during pruning. Invalid parent types and…
- kubectl wait can abort when a resource's status contains a malformed condition entry, even when other entries can describe the requested…
- Image and selector updates can fail for supported requests, leave selected resources unchanged, or report misleading results. Apply the…
- kubectl api-resources returns incorrect results for namespace scope, verb, and category filters. Matching discovered resources should be…
- kubectl create clusterrole --aggregation-rule can panic rather than generating a role with the requested aggregation selector. CronJob…
- Image updates can mishandle supported selections and option combinations, and local resource operations can use an unexpected namespace…
- Config views can show an unexpected context or accept unsupported arguments. ClusterRole, CronJob, and token creation can also mishandle…
- Apply requests can proceed with incompatible options or produce misleading feedback. Config views may not honor the selected context, and…
- Restricted debug sessions can create containers without the profile's expected protections. Edit sessions can also display misleading…
- Image-update requests can accept unsupported combinations or produce inconsistent feedback for container selections. Related local…
- Failed cordon/drain operations can leave the caller's node state inconsistent with the server. Drain also fails to recognize eviction…
- Updating resource limits for a selected container can fail even when that container is present. Environment updates from supported input…
- kubectl describe produces unreadable or incomplete annotation output when values are long or contain multiple lines. Preserve the…
- kubectl top pod can report success despite metrics being unavailable, or give inconsistent outcomes for named pods and empty selections…
187 more tasks.