AI Technology Brief
Why AI gives different answers from the same project files
The visible project folder is only the beginning of the evidence path. Different file revisions, permissions, OCR, chunking, retrieval, context, prompts, model versions, and sampling conditions can all change the answer. Reliable construction AI requires the complete path to be controlled and inspectable.
01 / What happened
Project-file AI moved from document demonstration to retrieval system.
Modern systems can extract project documents, divide them into retrievable units, search semantically, rerank evidence, generate answers, and attach citations. That architecture is useful, but every stage introduces a condition that can change which evidence reaches the model and how the answer is produced.
02 / Independently verifiable claims
Different answers can begin before generation starts.
- OpenAI documents that matching seed, prompt, model, temperature, and other parameters can improve reproducibility, but does not guarantee identical output.
- Microsoft and Google document that chunk size, overlap, retrieval count, filters, ranking, and reranking change which evidence reaches a model.
- Microsoft documents that document-level permissions and source changes must be represented correctly in the index; an index is not automatically identical to the visible folder.
- Anthropic documents source citations and long-context techniques, while also showing that information position and context construction can affect recall.
- Autodesk and Procore document revision, version, search-scope, metadata, and permission behavior that can change which project files a user can find or access.
- NIST identifies confabulation, provenance, source verification, empirical evaluation, human oversight, and post-deployment monitoring as material controls.
Lower temperature or a fixed seed can reduce one source of output variation. It cannot repair a stale revision, missing permission, incorrect OCR, weak chunk, omitted source, or retrieval result that never reached the model.
03 / The eight variation points
Diagnose the evidence chain before blaming randomness.
1. File identity
The files may share a visible name while fingerprints, source locations, attachments, or duplicate copies differ.
2. Revision and index state
One run may use the current revision while another uses a superseded, stale, orphaned, or partially indexed version.
3. Permissions
Two users can ask the same question but retrieve different evidence because folder, document, tenant, project, or sensitivity permissions differ.
4. Parsing and OCR
Scans, tables, drawings, reading order, handwriting, page boundaries, and extraction versions can produce different text and structure.
5. Chunking
Chunk size, overlap, headings, table treatment, and semantic boundaries decide which facts remain together.
6. Retrieval and ranking
Keyword, vector, hybrid, reranking, filters, top-k, thresholds, and query rewriting change the evidence selected.
7. Prompt and context
Instructions, conversation history, document order, context limits, and citations change what the model sees and prioritizes.
8. Model generation
Model snapshot, provider backend, temperature, sampling, tool behavior, and inherent nondeterminism can change wording or conclusions.
04 / Normal construction question
The same wording can query a different evidence set.
A project manager, estimator, document controller, and external consultant may type the same question. Their approved access, search scope, current revisions, and conversation history may not be the same.
User A
Current drawings + RFIs
Answer uses Rev C and approved change
User B
Specifications only
Answer misses the project-specific RFI
User C
All revisions + no current filter
Answer may retrieve superseded evidence
05 / Evidence architecture
A dependable answer needs a recorded path from source to claim.
Source registry
Identity, revision, status, permission
Extraction
OCR, layout, tables, text spans
Chunk index
Boundaries, headings, embeddings, ACLs
Retrieval
Query, filters, ranks, selected chunks
Context
Instructions, history, evidence order
Generation
Model, parameters, structured answer
Citation
Claim-to-passage support
Evaluation
Ground truth, review, acceptance
06 / Required operating records
Reproduce the run, not just the final wording.
Source document
Revision
Permission
Extraction
Chunk
Retrieval run
Generation run
Evaluation
07 / Versions, permissions and index state
The visible common data environment and the retrieval index can disagree.
Current evidence
Validate document status, revision order, effective date, superseded relationships, source authority, and latest approved state.
Permission-trimmed evidence
Apply source-aligned access to every indexed chunk and test which evidence each role can retrieve.
Reconciled index
Track source changes, deleted files, failed ingestion, orphaned chunks, indexing checkpoints, and full source-to-index counts.
08 / OCR and chunking
The model cannot retrieve evidence that parsing separated from its meaning.
Drawings, schedules, tables, forms, specifications, scanned correspondence, and mixed page layouts need different extraction and chunking treatment. Universal chunk settings are starting points, not construction acceptance criteria.
Parsing questions
- Was the page scanned or text-native?
- Were tables, headers, footnotes, and callouts preserved?
- Was reading order correct across columns and pages?
- Are drawing numbers, revisions, and references linked?
- Which parser and API version produced the text?
Chunking questions
- Does the chunk retain its document and section identity?
- Are headings and requirements kept with their clauses?
- Are table rows kept with column headers?
- Does overlap preserve cross-boundary meaning without flooding retrieval?
- Was the configuration evaluated on representative project questions?
09 / Retrieval and ranking
The answer is bounded by the evidence selected for context.
Query
Original wording, rewrite, expansion, project filters
Candidates
Keyword, vector, hybrid, top-k, threshold
Reranking
Ranker version, score, weighting, candidate count
Context
Selected chunks, ordering, truncation, citation metadata
A high retrieval score is not proof that the passage is controlling, current, complete, or sufficient for the business conclusion. Retrieval relevance and project authority are separate tests.
10 / Prompt, model and answer conditions
Control generation after controlling evidence.
Generation configuration
Model snapshot, provider backend fingerprint, seed where supported, temperature, top-p, tool definitions, response schema, and token limits.
Answer contract
Use only retrieved evidence; cite each material claim; distinguish confirmed, derived, uncertain, conflicting, and unavailable; abstain when evidence is insufficient.
Conversation state
System instructions, prior questions, prior tool results, user corrections, cached context, and document ordering can all change the next answer.
Human interpretation
Commercial, contractual, professional, safety, legal, programme, and approval conclusions remain with qualified and authorized people.
11 / Drawing and RFI example
One missing project-specific change can reverse the answer.
Question
Does the current electrical package require fire-rated cable supports in the east riser?
Run A evidence
Current drawing Rev C, approved RFI response, and specification section are indexed and permission-visible.
Run A answer
Yes, with citations to the current drawing detail and the approved RFI that changes the installation.
Run B evidence
The index still contains Rev B; the RFI folder is outside the user's permission scope; only the specification is retrieved.
Run B answer
The specification requirement is summarized, but the answer misses the approved project-specific change.
12 / RFQ deadline example
A correct answer needs the controlling invitation and every applicable addendum.
Question
When is the bid due, and which addenda must be acknowledged?
Run A
Retrieval includes the invitation email, Addendum 2, and the revised bid form. It returns the changed date and two required acknowledgements.
Run B
The email body and original instruction are retrieved, but the revised form falls below the retrieval cutoff. It returns the original deadline.
Control
Deadline and addenda are governed fields extracted from identified source versions, validated deterministically, and approved before use.
13 / Deterministic controls
Make the evidence boundary more stable than the generated wording.
Identity
Reject files without project, document, source, revision, status, and fingerprint identity.
Freshness
Record indexing checkpoints, source changes, deletions, superseded versions, and reconciliation status.
Permissions
Apply source-aligned ACLs to every retrievable chunk and test access with representative roles.
Provenance
Retain page, section, table, text span, chunk ID, source URL, and retrieval-run evidence.
Configuration
Version parsers, OCR, chunking, embeddings, indexes, retrieval, prompts, models, and output schemas.
Abstention
Return insufficient evidence when required sources are missing, conflicting, stale, or below acceptance thresholds.
14 / Failure analysis
A citation is useful only when the evidence is current, authorized, and supportive.
False sameness
Users see the same folder name but not the same files, versions, permissions, or index snapshot.
Superseded evidence
An older drawing or specification ranks above the current approved document.
Missing change
A deletion, revision, RFI, addendum, or updated attachment has not propagated to the index.
Broken structure
OCR or chunking separates a table header, note, drawing reference, qualification, or exception from its meaning.
Citation theatre
The answer includes a citation, but the cited passage does not support the material claim.
Overconfident synthesis
The model combines individually accurate passages into an unsupported project conclusion.
Permission drift
Indexed chunks are visible beyond or below the intended project and document access boundary.
Evaluation drift
A model, ranker, parser, index, prompt, or source change alters behavior without a regression test.
15 / Deployment and operating cost
Consistency has an infrastructure and review cost.
Direct long-context
Selected documents are supplied directly to the model.
Fewer indexing components, but document selection, context limits, position, permissions, token cost, and repeated processing still matter.
Managed retrieval
A hosted search or RAG service parses, chunks, indexes, retrieves, and optionally reranks evidence.
Faster infrastructure setup, but configuration, access control, provider changes, usage cost, and source reconciliation remain application responsibilities.
Client-controlled retrieval
Parsing, indexes, retrieval, run records, and evaluation operate in client-managed infrastructure.
More control over data and configuration, with higher engineering, compute, security, monitoring, backup, and maintenance responsibility.
Hybrid
Sensitive source identity and permissions remain client-controlled while approved extraction, retrieval, or generation services are used selectively.
Can separate trust zones, but version synchronization, networking, logs, failure ownership, and support boundaries become more involved.
- Source connectors and change detection
- OCR, layout extraction, and table processing
- Chunking, embeddings, indexing, and storage
- Search, reranking, query rewriting, and retrieval
- Model input, output, caching, and repeated runs
- Permission enforcement, secrets, logs, monitoring, and backup
- Evaluation datasets, automated graders, and domain review
- Human correction, exceptions, maintenance, and incident response
Complete cost per accepted answer = ingestion + extraction + indexing + retrieval + generation + evaluation + human review + exception handling
16 / Evaluation
Measure evidence retrieval separately from answer generation.
- Source coverage: required current documents available and permission-visible
- Retrieval recall: expected supporting chunks present in the retrieved set
- Retrieval precision: retrieved chunks materially relevant to the question
- Revision accuracy: current and controlling evidence selected
- Groundedness: material claims supported by retrieved evidence
- Citation precision: cited passage supports the claim it is attached to
- Citation coverage: material claims include inspectable evidence
- Answer correctness: reviewed outcome matches project ground truth
- Abstention quality: insufficient or conflicting evidence is surfaced instead of guessed
- Repeatability: outcome remains within accepted variance across repeated controlled runs
- Reviewer effort: correction time and severity by failure type
- Complete cost: ingestion, retrieval, generation, evaluation, review, and exception handling per accepted answer
Exact wording does not need to be identical when the meaning, evidence, caveats, and decision boundary remain correct. A materially different answer is unacceptable when it comes from missed controlling evidence, unsupported synthesis, invalid citation, or failure to abstain.
17 / Controlled pilot
Test deliberate variation before trusting everyday use.
- 01
Scope
One project question family, named source systems, required documents, user roles, and decision boundary.
- 02
Ground truth
Representative questions with approved answers, controlling revisions, accepted passages, and known conflicts.
- 03
Traceability
Every answer run records source snapshot, permissions, extraction, chunks, retrieval, prompt, model, citations, and reviewer.
- 04
Comparison
Repeat each question across controlled runs and intentional changes to permissions, revisions, retrieval, and model configuration.
- 05
Acceptance
Set thresholds for retrieval recall, correctness, groundedness, citation validity, abstention, critical failures, review time, and cost.
- 06
Change control
Re-run evaluation before promoting a new parser, embedding, index, ranker, prompt, model, source connector, or permission policy.
18 / StructuredLayer recommendation
Do not ask whether the AI gives the same sentence every time. Ask whether it used the same controlling evidence, respected the same permissions, preserved the same decision boundary, cited the claims it made, exposed uncertainty, and remained within accepted evaluation thresholds.
For consequential project questions, freeze and record the source snapshot, current revisions, user permissions, parser, chunking, retrieval, prompt, model, parameters, citations, and reviewer outcome. When required evidence is missing or conflicting, the correct answer is an owned exception, not a confident guess.
19 / Primary sources
Retrieval, citation, version, permission, and evaluation claims remain inspectable.
OpenAI Cookbook
Reproducible outputs with the seed parameter
OpenAI API documentation
Model endpoint compatibility
Microsoft Azure AI Search
Chunk documents for vector search
Microsoft Azure AI Search
Semantic ranking overview
Microsoft Azure AI Search
Security best practices
Microsoft Azure AI Search
Changed and deleted blobs
Microsoft Azure Document Intelligence
Document layout analysis
Google Cloud Vertex AI
Retrieval and ranking
Google Cloud Vertex AI
GenerateContentResponse grounding metadata
Google Cloud
Check grounding with RAG
Anthropic documentation
Citations
Anthropic
Prompt engineering for long context
Autodesk Docs documentation
Work with documents
Procore documentation
Search and filter documents
NIST
Generative AI Profile
Sources reviewed 21 July 2026. Product behavior, models, APIs, ranking systems, pricing, and documentation can change; verify current sources before implementation.