A Neural Network Based Teleoperation for Remote Controlled Vehicles

1. Introduction: The Challenges of Direct Teleoperation

As autonomous vehicle architectures advance toward broader deployment, state regulatory frameworks across several U.S. jurisdictions explicitly mandate remote vehicle teleoperation as an essential human-in-the-loop safety fallback. Direct teleoperation grants a remote human operator continuous steering, throttle, and braking authority when onboard autonomous software encounters complex edge cases outside its operational design domain (ODD). However, exercising direct real-time control over commercial cellular networks introduces severe control-theoretic vulnerabilities. Researchers Ning Ding and Azim Eskandarian highlight that direct teleoperation is fundamental constrained by two concurrent bottlenecks: stochastic communication latency over public wireless networks and the remote operator’s physical inability to perceive unmodeled environmental disturbances—such as crosswinds, road bank angles, grade shifts, and highly nonlinear tire-road dynamics.

Core Thesis: To maintain closed-loop stability across delay-prone wireless links without compromising real-time execution, the authors propose a dual-layer control architecture combining a unilateral Wave Variable (WV) transmission framework with a single-layer Radial Basis Function Network (RBFN). The WV framework passively guarantees stability under stochastic network delays without reflecting disruptive force feedback to the operator, while the RBFN actively estimates and cancels unmodeled dynamic disturbances online within strict edge-hardware compute budgets.

---

2. Dual-Layer Control Architecture: Wave Variables + Adaptive RBFN

The framework developed by Ding and Eskandarian addresses teleoperation instability by structurally decoupling delay-induced communication passivity from dynamic disturbance rejection on the vehicle side.

+------------------+         Wave Variable Channel         +--------------------+
| Remote Operator  | =====================================> |   Remote Vehicle   |
| Command Generator| <===================================== | State Feedback (v,)|
+------------------+   Forward Delay T1 / Backward Delay T2 +---------+----------+
                                                                      |
                                                                      v
                                                            +-------------------+
                                                            |   Adaptive RBFN   |
                                                            | Disturbance Cancel|
                                                            +-------------------+

2.1 Passive Signal Transmission via Unilateral Wave Variables

In traditional bilateral robotic teleoperation (e.g., surgical or industrial manipulators), Wave Variables (WV) passivate delay-prone communication channels by encoding force and velocity signals into wave vectors. However, directly applying bilateral formulations to ground vehicles is inappropriate: transmitting ground-reaction forces or steering-torque feedback back to the operator station injects high-frequency road vibrations and environmental disturbances directly into the operator’s interface.

The authors resolve this by formulating a vehicle-specific, unilateral Wave Variable architecture. Power-conjugate command and state pairs are defined across two independent transmission channels:

• Longitudinal Channel: Transmits command force $F^*$ and longitudinal velocity state $v^*$, defined by $(x_1^*, y_1^*) = (F^*, v^*)$.
• Lateral Channel: Transmits steering rate command $\dot{\delta}^*$ and vehicle yaw angle state $\varphi^*$, defined by $(x_2^*, y_2^*) = (\dot{\delta}^*, \varphi^*)$.

For each channel $i \in \{1, 2\}$, command and state variables are encoded into wave variables $u_i^*$ and $w_i^*$ using tuned characteristic channel impedances $k_i > 0$: $u_i^* = \frac{k_i x_i^* + y_i^*}{\sqrt{2 k_i}}, \quad w_i^* = \frac{k_i x_i^* - y_i^*}{\sqrt{2 k_i}}$

Subject to time-varying forward delay $T_1(t)$ and backward delay $T_2(t)$, wave transformations across the channel satisfy $u_{fi}(t) = u_{li}(t - T_1)$ and $w_{li}(t) = w_{fi}(t - T_2)$, where subscript $l$ denotes the operator (leader) and $f$ denotes the remote vehicle (follower). Decoding these wave signals at the vehicle recovers command force $F_f$ and steering rate $\dot{\delta}_f$, while returning state feedback $v_l$ and $\varphi_l$ to the human operator. This formulation guarantees channel passivity without reflecting external physical road forces back to the steering wheel.

2.2 Decoupled RBFN Dynamics and Real-Time Edge Adaptation

Because passive wave transmission preserves stability but cannot actively compensate for unmodeled physical forces (e.g., side gusts or road incline changes), an online adaptive Radial Basis Function Network (RBFN) operates directly on the vehicle’s embedded controller.

• Wave Variable Passivity: Passively absorbs stochastic network delays ($T_1, T_2$) without requiring an explicit mathematical delay model or predictor.
• RBFN Uncertainty Estimation: A single-layer Gaussian RBFN acts as a model-free online lumped uncertainty estimator. Single-layer networks are selected over deep multi-layer neural networks or model-predictive formulations (MPC/NMPC) to eliminate multi-layer backpropagation overhead and solver iteration delays.

To achieve strict real-time execution, the Gaussian basis functions use uniform center grids $c_j$ placed over an estimated operating envelope. Basis widths $b_j$ are derived via the neighbor-overlap parameterization rule: $b_j = \frac{d_{eff}}{\sqrt{\ln(1/\rho)}}$ where $d_{eff}$ represents the grid spacing and $\rho = 0.5$ sets the neighbor-overlap parameter, ensuring basis functions decay to half peak value at adjacent grid centers.

To prevent computational cross-coupling across control axes, the 6-DOF composite error vector $E = [E_1^T, E_2^T]^T \in \mathbb{R}^6$ is decoupled into longitudinal tracking error $E_1 = [e_1, \dot{e}_1]^T$ and lateral/yaw tracking error $E_2 = [e_2, \dot{e}_2, e_3, \dot{e}_3]^T$. The Lyapunov-based weight adaptation laws execute independently on edge hardware without cross-coupling calculations: $\dot{W}_1 = -\gamma E_1^T P_1 B h(E_1)$ $\dot{W}_2 = \eta h(E_2) E_2^T P_2 L$ where $\gamma, \eta > 0$ are adaptation gains, $P_1, P_2 \succ 0$ are symmetric positive-definite matrices derived from Lyapunov equations, $B = 1/M$ encapsulates vehicle mass, $L = D^\dagger$ is the pseudo-inverse of the lateral control input matrix, and $h(\cdot)$ is the vector of Gaussian basis functions.

2.3 Stability Bound Summary

Ding and Eskandarian establish closed-loop stability using a composite Lyapunov storage function combining human operator passivity, wave-channel energy storage, and vehicle tracking error dynamics: $S = S_{op} + S_{WV} + V_{RV}$ The wave storage function $S_{WV}(t) = \frac{1}{2} \sum_{i=1}^2 \left( \int_{t-T_1}^t u_{li}^2 d\tau + \int_{t-T_2}^t w_{fi}^2 d\tau \right)$ explicitly captures power stored in the communication pipeline. Differentiating $S$ demonstrates that time-varying transmission delays $T_1, T_2$ enter the system differential equations strictly through integration bounds. Under bounded operator inputs ($\|x_f^T y_f^*\| \le \mu_0$ and $\|C^T x_f\| \le \mu_1$), the composite Lyapunov derivative satisfies: $\dot{S} \le -\frac{\alpha_0}{2} \|E\|^2 + \frac{\mu_1^2}{2\alpha_0} + \mu_0$ This proves that composite tracking errors $E$ remain Uniformly Ultimately Bounded (UUB). The residual ultimate error bound $R$ is defined as: $R := \frac{\mu_1^2}{\alpha_0^2} + \frac{2\mu_0}{\alpha_0}$ Because $R$ depends strictly on operator command bounds ($\mu_0, \mu_1$) and positive design gains ($\alpha_0$), the system’s ultimate stability envelope is proven to be independent of network delay durations $T_1$ and $T_2$.

---

3. Simulation Benchmarks: Comparative Performance Trade-Offs

The proposed controller was evaluated in a high-fidelity MATLAB/Simulink environment utilizing a 3-DOF nonlinear vehicle dynamic model (incorporating longitudinal, lateral, and yaw motions alongside aerodynamic drag, rolling resistance, grade effects, and bank angles). All algorithms were evaluated under a strict $20\text{ ms}$ sampling budget ($50\text{ Hz}$), enforcing zero-order holds whenever controller execution time ($t_E$) exceeded $20\text{ ms}$. Stochastic wireless latency was generated using a Generalized Extreme Value (GEV) distribution fitted to empirical 4G network traces.

Execution Runtime Budget vs. Controller Execution Time (t_E)
20 ms Edge Budget Threshold |-----------------------------------|
NMPC Lateral (75.20 ms)    |===================================================> [OVERRUN]
NMPC Long. (44.38 ms)      |===============================> [OVERRUN]
MPC Lateral (2.58 ms)      |===>
LQR Lateral (0.12 ms)      |=
RBFN Proposed (0.03 ms)    |*
PID Long. (0.02 ms)        |*

The adaptive RBFN architecture was benchmarked against Proportional-Integral-Derivative (PID), Linear Quadratic Regulator (LQR), Model Predictive Control (MPC), and Nonlinear Model Predictive Control (NMPC) frameworks across severe environmental disturbances ($8^\circ$ road slope, $10\text{ m/s}$ crosswinds, $8^\circ$ bank angle).

Controller	Longitudinal $t_E$ (ms)	Lateral $t_E$ (ms)	Key Performance Metrics / Limitations Under Disturbances
RBFN (Proposed)	~0.03 ms	~0.03 ms	Fast recovery ($t_2=8.8\text{ s}$ at $120\text{ km/h}$), ultra-low runtime, online adaptation.
PID	0.01–0.02 ms	N/A	Lowest runtime, but severe settling degradation ($t_2=48.9\text{ s}$) under slope/wind.
LQR	N/A	~0.11–0.12 ms	Low latency, satisfactory nominal tracking, lacks online disturbance adaptation.
MPC	1.79–2.01 ms	2.40–2.58 ms	Moderate compute, but degrades ($t_2=10.8\text{ s}$) when disturbances exceed internal model bounds.
NMPC	39.56–44.38 ms	72.40–75.20 ms	High model accuracy, but $t_E \gg 20\text{ ms}$ budget creates massive control overruns.

Analysis of Simulation Trade-Offs

• Computational Efficiency & Edge Real-Time Compliance: The RBFN controller executes in approximately $0.03\text{ ms}$, matching the speed profile of basic PID ($0.01\text{--}0.02\text{ ms}$) and LQR ($0.11\text{--}0.12\text{ ms}$). Conversely, NMPC requires $39.56\text{--}44.38\text{ ms}$ for longitudinal control and $72.40\text{--}75.20\text{ ms}$ for lateral control per step. On vehicle edge processors operating on a $20\text{ ms}$ loop, NMPC triggers continuous actuator command holds, inducing lag and control degradation.
• Quantitative Recovery Under Severe Disturbances: In longitudinal evaluations at $120\text{ km/h}$ under an $8^\circ$ slope and $10\text{ m/s}$ wind gust, fixed-gain PID required $48.9\text{ s}$ to settle (peak speed error $h_2 = 3.2\text{ km/h}$) due to its inability to update internal dynamic parameters. Kinematic MPC required $10.8\text{ s}$ ($h_2 = 1.5\text{ km/h}$) as environmental inputs fell outside its predictive model horizon. The proposed adaptive RBFN compensated for unmodeled drag and grade online, recovering nominal speed in $8.8\text{ s}$ ($h_2 = 1.9\text{ km/h}$) while using less than $2\%$ of MPC’s compute time.
• Simulated Delay Sensitivity (Tables II & IV): Simulated forward delays were escalated from $300\text{ ms} \to 400\text{ ms} \to 500\text{ ms}$ at $60\text{ km/h}$. Under longitudinal slope/wind disturbances, RBFN peak speed deviations ($h_2$) degraded gracefully from $2.1\text{ km/h} \to 2.3\text{ km/h} \to 2.6\text{ km/h}$ (settling times $t_2$ rising from $8.6\text{ s} \to 8.8\text{ s} \to 10.4\text{ s}$). Under lateral bank disturbances, deviation ($l_1$) scaled mildly from $0.36\text{ m} \to 0.38\text{ m} \to 0.39\text{ m}$ ($t_1$ increasing from $19.1\text{ s} \to 22.1\text{ s} \to 24.5\text{ s}$), demonstrating stability prior to hardware deployment.
• ISO 3888-1 Double Lane-Change Benchmarks: Evaluated under aggressive evasive double lane changes at $100\text{ km/h}$ and $130\text{ km/h}$ (Table V), RBFN limited lateral deviations to $l_1 = 0.21\text{ m}, l_2 = 0.22\text{ m}$ at $100\text{ km/h}$, and $l_1 = 0.23\text{ m}, l_2 = 0.24\text{ m}$ at $130\text{ km/h}$. RBFN outperformed LQR ($0.27\text{ m} / 0.30\text{ m}$), MPC ($0.26\text{ m} / 0.30\text{ m}$), and NMPC ($0.26\text{ m} / 0.29\text{ m}$).

---

4. Real-World Hardware-in-the-Loop (HIL) 4G Testing

To validate the theoretical architecture against real human operator dynamics and live cellular network behavior, Ding and Eskandarian performed hardware-in-the-loop (HIL) experiments on a wireless physical testbed.

4.1 Testbed & Delay Envelope

• Operator Control Station: Integrated a Logitech G29 gaming steering wheel and pedal set with a ROS2 (Foxy) node executing at a $20\text{ ms}$ cycle time ($50\text{ Hz}$). Onboard vehicle camera streams provided visual feedback.
• Physical Remote Vehicle: A 1/10th-scale test vehicle equipped with a Vedder Electronic Speed Controller (VESC) providing direct motor torque regulation.
• Cellular Network Pipeline: Evaluated over an active 4G network with measured default latencies of ~100 ms forward (operator-to-vehicle) and ~300 ms reverse (vehicle-to-operator). Latency stress tests emulated forward delays up to 500 ms.

4.2 Safety Failure Prevention (Lane Containment)

Physical experiments exposed critical safety limits during high-latency lateral maneuvers on banked surfaces ($5^\circ$ slope). Safety boundaries were quantified by measuring distances from vehicle wheels to lane boundaries ($W_l, W_r$), where negative values ($W_l < 0$) indicate outer/inner wheel lane crossing failures.

• Unassisted Direct Control Breach: At $1.20\text{ m/s}$ forward velocity under $500\text{ ms}$ delay on a banked road (Table VII), unassisted direct human teleoperation suffered complete loss of lane containment. The inner wheels breached the lane boundary, registering $W_l = -0.15\text{ m}$ in laboratory and outdoor downhill tests, and $W_l = -0.10\text{ m}$ in outdoor uphill tests.
• Assisted System Containment: With the proposed neural-wave controller active, the vehicle maintained positive inner-wheel safety margins across all elevated latency levels up to $500\text{ ms}$. At $1.20\text{ m/s}$ under $500\text{ ms}$ delay, the system achieved positive margins of $W_l = +0.22\text{ m}$ (laboratory), $W_l = +0.27\text{ m}$ (outdoor uphill), and $W_l = +0.45\text{ m}$ (outdoor downhill). The assisted controller maintained $W_l \ge 0.20\text{ m}$ across all physical experimental configurations.

4.3 Trajectory Tracking

Empirical data collected across laboratory and outdoor environments ($5\text{ m/s}$ crosswinds, uneven terrain) confirmed robust trajectory execution under latency stress:

• Target Speed Tracking Stability: As documented in Table VI, at $500\text{ ms}$ latency and $2.0\text{ m/s}$ target speed outdoors, speed deviation without assistance reached $1.3\text{ m/s}$ ($65\%$ error margin). The adaptive RBFN suppressed speed deviations to $0.6\text{ m/s}$. Speed traces in Figure 2 show that while unassisted direct control overshot $2.0\text{ m/s}$ targets up to $2.6\text{ m/s}$, the RBFN system maintained tight velocity tracking near $2.0\text{ m/s}$.
• Closed-Track Speed Regulation: Experimental closed-track speed traces in Figure 3 ($v_d$ vs. $v_f$) demonstrate smooth velocity tracking. In variable-speed trials ($[0.9, 1.2]\text{ m/s}$), the RBFN closely matched desired velocity $v_d$ generated by the Trajectory Design Module, whereas unassisted driving produced large velocity oscillations.
• Track Boundary Maintenance: Recorded closed-loop spatial trajectories in Figure 4 confirm that the Trajectory Design Module generates dynamically feasible reference states in real time. Steering execution remained centered within track boundaries during both idle-speed ($0.65\text{ m/s}$) and variable-speed ($[0.9, 1.2]\text{ m/s}$) tests.
• Latency Robustness Limits: Closed-loop path trajectories in Figure 5 illustrate spatial paths under extended delays ($300\text{ ms}, 400\text{ ms}, 500\text{ ms}$). Even at $500\text{ ms}$ network latency, the vehicle maintained stable path bounds without destabilizing oscillations.

---

5. Failure-First Research Takeaways & Conclusion

For robotics engineers, red-teamers, and AI safety researchers evaluating edge deployment constraints in safety-critical autonomous systems, this empirical paper highlights key systemic insights:

• Edge Compute Bottlenecks Can Exceed Network Latency Risks: Model-heavy non-linear predictive controllers (NMPC) introduce internal execution delays ($>40\text{--}75\text{ ms}$) that exceed real-time edge processing budgets ($20\text{ ms}$). This compute overrun induces actuator command holds that degrade vehicle controllability independently of network performance.
• Structural Decoupling Protects Against Stochastic Jitter: Separating passivity preservation (via Wave Variables) from online disturbance compensation (via single-layer RBFNs) provides a resilient safety architecture. Passive transmission channels handle network latency jitter, allowing lightweight neural networks to focus strictly on physical disturbance rejection.
• Online Neural Adaptation Prevents Physical Boundary Failures: Unassisted teleoperation degrades rapidly when latency is coupled with unmodeled environmental forces, leading to boundary breaches. Low-dimensional, single-layer neural adaptation prevents vehicle control loss during unmodeled friction or incline changes where fixed-gain controllers fail.

In conclusion, Ning Ding and Azim Eskandarian demonstrate that combining unilateral wave variable passivity with decoupled adaptive RBFNs resolves the dual challenges of network latency and dynamic environmental uncertainty in ground vehicle teleoperation. The authors note that future research will focus on extending theoretical stability proofs to coupled multi-input multi-output (MIMO) vehicle dynamics, developing dedicated passivity-preserving mechanisms for severe network packet loss, and validating the architecture on full-scale autonomous vehicle platforms.

Read the full paper on arXiv · PDF