1. Meeting Summary
I met with Karsten and Jonathon to work through the SAP recovery process following a system corruption caused by a transport into the finance area of P10. The system was restored to approximately 10:30 European time, which means all postings and warehouse transactions completed after that point were lost from the restored environment and now need to be reconstructed.
The core issue is that outbound deliveries, pick tickets, packing records, and goods issue postings created or completed during the affected window now no longer align with what physically happened in the warehouse. We walked through the different transaction states and what each one means operationally. The simplest cases are deliveries created with no downstream activity, since those can be recreated directly. The more complex cases involve pick tickets that were printed, physically picked, confirmed, partially packed, fully packed, or already shipped. For shipments that have already left the building, the physical activity cannot be redone, so the recovery approach will require creating new system documents and maintaining a mapping between the old document numbers on the paperwork and the newly recreated numbers in SAP.
I clarified that from the warehouse side we had several transaction states in play at the time of the restore, including orders picked but not confirmed, picked and confirmed but not packed, partially packed, and fully packed but not yet through UPS scaling. We also discussed the fact that some packages have already been scanned by UPS, which means we likely will not be able to produce a fully accurate manifest for those shipments. There may be downstream freight charge or exception handling implications, but there is no practical way to reverse those physical shipments.
The immediate operating plan is to first identify the physical status of affected pick tickets in the warehouse, especially separating tickets that were printed but never physically picked from those that were actually picked but not confirmed in the system. Karsten's team is preparing the supporting lists, including the pick ticket list and parcel list, while Uli is expected to recreate deliveries and transport orders in SAP, using the same source bins where possible so the new paperwork aligns with what was physically picked. On our side, we will begin marking physical status in the shared restore activities file so the rebuild effort can be prioritized and executed accurately.
2. Attendee List
- Karsten Todt
- Steve Wade
- Jonathon Rosenberger
3. Action Items
- [Karsten Todt] Prepare and maintain the pick ticket recovery list for transactions created or confirmed during the affected time window.
- [Karsten Todt] Add or prepare the parcel list, including outbound delivery information, to support identification of packed shipments.
- [Karsten Todt] Coordinate with Uli on recovery status and confirm when recreated deliveries and transport orders are ready.
- [Karsten Todt / Uli] Recreate affected outbound deliveries in SAP after material movements are canceled and stock is aligned.
- [Karsten Todt / Uli] Recreate transport orders for affected picks, using the original bins where possible to keep warehouse execution aligned.
- [Karsten Todt / Uli] Print or make available newly recreated pick tickets so the warehouse can match them to physical picks.
- [Steve Wade / team] Identify all affected physical pick tickets in the warehouse and determine their actual status.
- [Steve Wade / team] Mark which pick tickets were printed but not physically picked.
- [Steve Wade / team] Mark which pick tickets were physically picked but not confirmed in the system.
- [Steve Wade / team] Collect untouched pick tickets and document them in a spreadsheet or in the shared file so HQ knows those do not require physical recovery handling.
- [Steve Wade / team] Use the old-to-new document mapping to match recreated paperwork to physical inventory and in-process orders.
- [Kelsey] Update the shared restore activities document with physical pick status for affected tickets.
- [Steve Wade / team] Review conveyor and packing areas for cartons that were packed but not fully shipped and determine whether relabeling and document replacement are possible.
- [Jonathon Rosenberger] Add an owner column to the shared tracking sheet so responsibilities can be split between HQ and North America.
4. Relevant Timelines
- System corruption occurred yesterday at approximately 10:40 European time.
- Issue was recognized at approximately 3:00 PM European time the same day.
- System was restored overnight to a timestamp of approximately 10:30 European time.
- All postings and warehouse transactions completed after the restore point must be reviewed and reconstructed.
- Immediate next step: warehouse team to begin identifying physical pick ticket status now.
- Immediate next step: Karsten to prepare and share the parcel list while the warehouse team works the pick ticket file.
- Follow-up trigger: Karsten's team will call back once they are further along and Uli is ready to begin or continue document recreation.
5. Additional Notes
- Approximately 590 pick tickets were identified as created or confirmed within the affected time window.
- That volume may be inflated by backlog and carryover activity from the previous day, including receipts that released backordered items and increased delivery count.
- The recovery challenge is not just system recreation. It is reconciliation between SAP's restored state and the actual physical warehouse state.
- The key distinction for operations is between system status and physical status. Systemically, some cases may look identical, but operationally they require very different recovery actions.
- For orders already shipped, we will need a clean cross-reference between original and recreated delivery, pick, and packing numbers because the paperwork and labels already on the boxes cannot be changed retroactively.
- UPS manifest accuracy will likely be impacted for shipments that already left the building before the restore issue was fully understood.
- Serial number data appears recoverable from the copied system, which should reduce risk on serialized inventory.
- The most practical near-term approach is to avoid re-slotting or returning physically picked product to bins unless absolutely necessary, since that would create significant additional labor and delay.