Shipshape is changing how we bring intelligence to the home. Our platform is going to be getting even smarter in 2025. We are adding conversational agents starting with our home health assessment and home health score systems that give homeowners insight into the status of their home Helping you keep up on maintenance and preventative repairs.

Soon you’ll be able to talk to Sam, our conversational agent about any question you might have about your home. Just like a handyman in your pocket.

Homeowners will be able to choose how they interact with Sam. Whether it’s over messages, Voice, or even using their camera. Talk to Sam directly from the app with built-in speech to text and text to speech. We’re needing to add information to the Shipshape platform such as documents or labels off of the water heater. Take a photo and our AI will extract out relevant information for your home health assessment and more. Simply take a photo of the label on a sump pump, HVAC, or water heater, and effortlessly let Shipshape extract and log the make model and serial number.

Shipshape is revolutionizing home maintenance with a cutting-edge platform that integrates IoT sensors, advanced AI capabilities, and a comprehensive 25-point Home Health Assessment. This paper delves into the technical underpinnings of our approach to Retrieval-Augmented Generation (RAG), Computer Vision (CV) for data extraction (including OCR), and our OpenAPI-based framework for seamless device integration. By harnessing these technologies, we aim to provide homeowners, installers, and contractors with actionable insights into a home's health and potential risks.

‍

System Architecture

IoT Ecosystem & Home Base Station

Smart Devices with Sensors: ShipShape integrates humidity, temperature, water presence, and power usage sensors placed in critical and difficult to access areas (e.g., basements, around water heaters, sump pumps, washers/dryers).

Base Station (Gateway): ShipShape offers a gateway and supports gateways from popular providers such as Nest and Honeywell. A local gateway device within the home manages data collection from the sensors placed strategically throughout the house via standard protocols (Wi-Fi, ZigBee, Z-Wave). Data is forwarded from the gateway to the ShipShape Cloud using encrypted communication.

OpenAPI Framework: ShipShape systems include OpenAPI endpoints that allow third-party services and devices to connect with the platform. This enables a future-proofed modular ecosystem where new sensors and home automation systems can be integrated efficiently.

Mobile Apps & User Interfaces

iOS & Android Apps: Built on a cross-platform framework, the mobile apps communicate with ShipShape's backend via secure REST APIs defined in our OpenAPI specification.

UX Considerations: A RAG-based agent guides customers through sensor setup, prompting them to capture device images and make/model details of the appliance using their smartphone camera. The streamlined UI ensures minimal friction during installation and activation.

Cloud Infrastructure

Data Ingestion & Storage: Sensor readings are stored in scalable, cloud-based data warehouses, ensuring robust data retention and quick retrieval.

Compute Layer: The SAM agent and Computer Vision models—run in containerized microservices, providing easy scalability and high availability.

‍

SAM Agent Reasoning

ShipShape AI uses a combination of top AI backends like OpenAI, Grok 3, Llama, Claude, to power our SAM reasoning engine. Our architecture uses an enhanced RAG (Retrieval-Augmented Generation) is a technique that combines large language models (LLMs) with a specialized knowledge base to generate more accurate, context-aware responses. ShipShape implements RAG as follows:

1. Privacy

• Our top consideration is privacy, making sure we securely isolate our customer’s home data using state of the art best practices to keep your data safe. This includes the use of industry leading authentication, encryption, monitoring, logging technology.

2. Knowledge Base Construction

• We aggregate all relevant content about your home from user manuals to best-practice guides and official documentation from appliance manufacturers—into a document store along with the real time data from the ShipShape home API.
• Metadata Tagging: Each document is tagged with appliance type, manufacturer, model number, and relevant topics (e.g., "dehumidifier troubleshooting," "HVAC best practices").

3. Query and Action Flow

• User Query: A homeowner or installer asks a question in natural language via the mobile app or web portal (e.g., "What does it mean when my sump pump cycles frequently?").
• Retrieval: The system uses semantic search to find the most relevant documents, extracting passages likely to contain the answer.
• Generation: A large language model (fine-tuned on home maintenance data) then synthesizes these passages into a coherent answer.
• Response & Context: The user receives an easy-to-understand explanation and references (e.g., "Based on the manufacturer's guidelines, your sump pump model usually cycles every 30 minutes if there's high groundwater…")

4. Reasoning & Recommendations

• Contextual Reasoning: The LLM is further enhanced with domain-specific reasoning modules, allowing it to interpret sensor readings in real time (e.g., humidity spiking in the basement may correlate with a sump pump issue).
• Actionable Insights: The system provides next-step recommendations—like calling a certified contractor or scheduling an HVAC inspection—backed by curated knowledge from both industry experts and user experiences.

‍

New UX Opportunities with SAM AI

Interactive Agents like SAM not only provides answers but can also proactively reach out when sensor anomalies occur.

Adaptive Onboarding: New users are walked through sensor placement and initial calibrations. If the user installs a new device, the agent automatically retrieves relevant calibration instructions from the knowledge base. 

Expert Assistance: In complex scenarios, the agent seamlessly escalates queries to human experts for a more in-depth consultation, preserving conversation context for a smooth handoff.

‍

New powers - Computer Vision & Home Health Assessment

ShipShape leverages Llama 3.2 Vision and Grok 3 for advanced image recognition and OCR capabilities:

1. Device Registration via OCR

Image Capture: Homeowners use the mobile app to photograph the nameplate or label on water heaters, sump pumps, dehumidifiers, washers, and other appliances.

OCR Analysis: The CV module extracts the make, model, and serial number from the image. This data is then cross-referenced with our knowledge base, automatically populating the device's maintenance schedule and known issues.

Meta-Tagging: Each device in the user's account is tagged with metadata (e.g., "Water Heater, AO Smith, 40-gallon, installed 2018"), which informs future RAG queries and proactive notifications.

2. Damage Detection & Risk Assessment

Visual Inspection: Users can upload images of potential problem areas (e.g., suspicious stains on the ceiling and rust on the boiler). The system analyzes these images for mold, leaks, or corrosion signs.

Comparative Analysis: The CV module can compare historical images of the same location to identify changes over time (e.g., a growing stain).

Real-Time Alerts: Once a risk is detected, users receive an in-app notification and an optional text message prompting immediate remediation steps.

‍

25-Point Home Health Assessment & Scoring

Our annual 25-Point Home Health Assessment combines IoT sensor data, Computer Vision findings, and manual user inputs to generate an actionable "Home Health Score." 

1. Data Collection

Sensor Parameters: Temperature, humidity, water presence, power usage, and any sensor-specific diagnostics.

Computer Vision: Analysis of uploaded images for structural or appliance-based issues.

Manual Inspection Items: The user or a professional might confirm items like "Check the circuit breaker for frequent tripping" or "Inspect the roof for missing shingles."

2. Scoring Methodology

Weighted Criteria: Each of the 25 points has a weight based on potential risk. For instance, identifying a sump pump issue might be more urgent than a slightly dirty air filter.

AI-driven Insights: The RAG system cross-references industry standards and manufacturer guidelines to assign severity levels to anomalies.

Aggregation: Scores from each point are combined into a single Home Health Score, ranging from "Excellent" (minimal issues) to "At Risk" (immediate action needed).

3. Actionable Outcome

Proactive Maintenance: Users receive a prioritized checklist for repairs or further inspection.

Insurance & Compliance: A well-documented Home Health Score can be shared with insurers to potentially lower premiums, showcasing proactive maintenance.

Longitudinal Tracking: Over time, historical scores and reports let homeowners see trends, enabling them to address minor issues before they escalate.

‍

Conclusion

ShipShape's technical ecosystem—encompassing RAG-based reasoning, advanced Computer Vision, an OpenAPI-driven architecture, and the 25-point Home Health Assessment—serves as the industry standard for proactive home maintenance. By integrating AI at each stage of the user journey, we empower homeowners to safeguard their properties, reduce insurance costs, and enjoy peace of mind knowing that they are backed by cutting-edge technology and expert support.

‍

For more information, please visit: shipshape.ai

Join us in building the future of smart home maintenance, where AI-driven insights, seamless IoT integration, and comprehensive health assessments converge to keep your home safe, healthy, and efficient.

‍