AR and Google ML Kit
Linkedin Instagram Facebook X-twitter

Introduction:

Augmented reality (AR) has transformed the gaming industry, offering players immersive experiences that blend the virtual and real worlds seamlessly. One of the key technologies driving AR gaming is object detection, which allows games to recognize and interact with real-world objects captured by a device’s camera. In this blog post, we’ll explore how object detection is used in game development, diving into a codebase that demonstrates its implementation.

Understanding Object Detection in Gaming:

Object detection involves identifying and locating specific objects within an image or video frame. In the context of gaming, object detection enables developers to create experiences where virtual objects are overlaid onto the real world, enhancing player interaction and immersion.
 

Exploring the Codebase:

We’ll dive into the flutter codebase that demonstrates the implementation of object detection in an AR gaming scenario. Here’s a breakdown of the key components:
 

Widget Initialization

The ARGameView widget is a stateful widget that takes a title and a callback function onDetectedObject. This function will be invoked whenever an object is detected in the AR game.
class ARGameView extends StatefulWidget {
  ARGameView({
    Key? key,
    required this.title,
    required this.onDetectedObject,
  }) : super(key: key);
  final String title;
  final Function(DetectedObject) onDetectedObject;

  @override
  State<ARGameView> createState() => _ARGameViewState();
}

State Management

The _ARGameViewState class manages the state of the ARGameView widget. It initializes the object detector and other necessary variables in the initState method.

class _ARGameViewState extends State<ARGameView> {
  ObjectDetector? _objectDetector;
  DetectionMode _mode = DetectionMode.stream;
  bool _canProcess = false;
  bool _isBusy = false;
  CustomPaint? _customPaint;
  String? _text;
  var _cameraLensDirection = CameraLensDirection.back;
  int _option = 0;
  final _options = {
    'default': '',
    'object_custom': 'object_labeler.tflite',
  };
  
  @override
  void initState() {
    super.initState();
    _initializeDetector();
  }

Detector Initialization

This code initializes an object detector using an existing machine learning model from Google ML Kit. objectDetector in this code snippet is a component of Google ML Kit’s computer vision capabilities,

allowing developers to integrate powerful object detection and classification functionalities into their application with ease. 

In the following block, it’s used for identifying and localizing objects within images or video frames. The model is capable of detecting multiple objects simultaneously and, optionally, classifying them based on predefined categories. 

void _initializeDetector() async {
    _objectDetector?.close();
    _objectDetector = null;

    if (_option == 0) {
      final options = ObjectDetectorOptions(
        mode: _mode,
        classifyObjects: true,
        multipleObjects: true,
      );
      _objectDetector = GoogleMlKit.vision.objectDetector(options);
    } else if (_option > 0 && _option <= _options.length) {
      final option = _options[_options.keys.toList()[_option]] ?? '';
      final modelPath = await getAssetPath('assets/ml/$option');
      final options = LocalObjectDetectorOptions(
        mode: _mode,
        modelPath: modelPath,
        classifyObjects: true,
        multipleObjects: true,
      );
      _objectDetector = GoogleMlKit.vision.objectDetector(options);
    }

    _canProcess = true;
}
 

Image Processing

The _processImage method employs the image recognition technique to analyze the captured image and detect objects using the initialized detector. Once objects are detected, the UI is updated accordingly using the _updateUI method.

Future<void> _processImage(InputImage inputImage) async {
    if (_objectDetector == null) return;
    if (!_canProcess) return;
    if (_isBusy) return;
    _isBusy = true;
    setState(() {
      _text = '';
    });
    final objects = await _objectDetector!.processImage(inputImage);
    _updateUI(objects);
    _isBusy = false;
    if (mounted) {
      setState(() {});
    }
}

UI Update

The _updateUI method updates the UI with the detected objects. If objects are detected, it displays the number of objects detected along with a visual representation of the objects using the CustomPaint widget. Otherwise, it displays a message indicating that no objects were detected.

void _updateUI(List<DetectedObject> objects) {
    if (objects.isNotEmpty) {
      setState(() {
        _text = 'Objects Detected: ${objects.length}';
        _customPaint = CustomPaint(
          painter: ObjectDetectPainter(objects),
        );
      });
    } else {
      setState(() {
        _text = 'No Objects Detected';
        _customPaint = null;
      });
    }
}
Here's how the final code looks like, 

class ARGameView extends StatefulWidget {

  ARGameView({
    Key? key,
    required this.title,
    required this.onDetectedObject,
  }) : super(key: key);

  final String title;
  final Function(DetectedObject) onDetectedObject;

  @override
  State<ARGameView> createState() => _ARGameViewState();
}

class _ARGameViewState extends State<ARGameView> {
  ObjectDetector? _objectDetector;
  DetectionMode _mode = DetectionMode.stream;
  bool _canProcess = false;
  bool _isBusy = false;
  CustomPaint? _customPaint;
  String? _text;
  var _cameraLensDirection = CameraLensDirection.back;
  int _option = 0;
  final _options = {
    'default': '',
    'object_custom': 'object_labeler.tflite',
  };

  @override
  void initState() {
    super.initState();
    _initializeDetector();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Stack(
        children: [
          DetectorView(
            title: 'AR Game Detector',
            customPaint: _customPaint,
            text: _text,
            onImage: _processImage,
            initialCameraLensDirection: _cameraLensDirection,
            onCameraLensDirectionChanged: (value) =>
                _cameraLensDirection = value,
            onCameraFeedReady: _initializeDetector,
            initialDetectionMode: DetectorViewMode.values[_mode.index],
            onDetectorViewModeChanged: _onScreenModeChanged,
          ),
          Positioned(
            top: 30,
            left: 100,
            right: 100,
            child: Row(
              children: [
                Spacer(),
                Container(
                  decoration: BoxDecoration(
                    color: Colors.black54,
                    borderRadius: BorderRadius.circular(10.0),
                  ),
                  child: Padding(
                    padding: const EdgeInsets.all(4.0),
                    child: _buildDropdown(),
                  ),
                ),
                Spacer(),
              ],
            ),
          ),
        ],
      ),
    );
  }

  Widget _buildDropdown() => DropdownButton<int>(
        value: _option,
        icon: const Icon(Icons.arrow_downward),
        elevation: 16,
        style: const TextStyle(color: Colors.blue),
        underline: Container(
          height: 2,
          color: Colors.blue,
        ),
        onChanged: (int? option) {
          if (option != null) {
            setState(() {
              _option = option;
              _initializeDetector();
            });
          }
        },
        items: List<int>.generate(_options.length, (i) => i)
            .map<DropdownMenuItem<int>>((option) {
          return DropdownMenuItem<int>(
            value: option,
            child: Text(_options.keys.toList()[option]),
          );
        }).toList(),
      );

  void _onScreenModeChanged(DetectorViewMode mode) {
    switch (mode) {
      case DetectorViewMode.gallery:
        _mode = DetectionMode.single;
        _initializeDetector();
        return;
      case DetectorViewMode.liveFeed:
        _mode = DetectionMode.stream;
        _initializeDetector();
        return;
    }
  }

  void _initializeDetector() async {
    _objectDetector?.close();
    _objectDetector = null;

    if (_option == 0) {
      final options = ObjectDetectorOptions(
        mode: _mode,
        classifyObjects: true,
        multipleObjects: true,
      );
      _objectDetector = GoogleMlKit.vision.objectDetector(options);
    } else if (_option > 0 && _option <= _options.length) {
      final option = _options[_options.keys.toList()[_option]] ?? '';
      final modelPath = await getAssetPath('assets/ml/$option');
      final options = LocalObjectDetectorOptions(
        mode: _mode,
        modelPath: modelPath,
        classifyObjects: true,
        multipleObjects: true,
      );
      _objectDetector = GoogleMlKit.vision.objectDetector(options);
    }

    _canProcess = true;
  }

  Future<void> _processImage(InputImage inputImage) async {
    if (_objectDetector == null) return;
    if (!_canProcess) return;
    if (_isBusy) return;
    _isBusy = true;
    setState(() {
      _text = '';
    });
    final objects = await _objectDetector!.processImage(inputImage);
    _updateUI(objects);
    _isBusy = false;
    if (mounted) {
      setState(() {});
    }
  }

  void _updateUI(List<DetectedObject> objects) {
    if (objects.isNotEmpty) {
      // Update UI with detected objects
      setState(() {
        _text = 'Objects Detected: ${objects.length}';
        _customPaint = CustomPaint(
          painter: ObjectDetectPainter(objects),
        );
      });
    } else {
      setState(() {
        _text = 'No Objects Detected';
        _customPaint = null;
      });
    }
  }
}

Use Cases in Gaming:

Integrating object detection in Mobile App Development for game development opens up a plethora of use cases and gameplay possibilities, leveraging Machine learning and Google ML Kit:

  1. 1. Augmented Reality Games: Players can immerse themselves in virtual adventures overlaid onto their surroundings, engaging in treasure hunts, creature hunts, or virtual battles, thus fostering collaboration and competition within the flutter community.
  3. 2. Object Recognition Challenges: Games can challenge players to identify and interact with real-world objects to unlock rewards, solve puzzles, or progress through levels, enhancing engagement and interactivity. This integration can be facilitated through a dedicated GitHub repository for easy access and collaboration among developers.
  5. 3. Immersive Storytelling: Object detection can enrich storytelling in games by triggering events or narrative elements based on real-world objects detected by the camera, offering personalized and interactive experiences for players, thus pushing the boundaries of mobile gaming experiences.
  7. 4. Multiplayer AR Experiences: Friends can collaborate or compete in multiplayer AR games, working together or against each other to achieve objectives or complete challenges within shared virtual environments, fostering social interaction and engagement in the gaming community.

Summary:

Object detection technology is revolutionizing the gaming industry, enabling developers to create immersive augmented reality experiences that blur the lines between the virtual and real worlds. By exploring the codebase and understanding its implementation, we've gained insight into how object detection can be leveraged to build innovative and engaging gaming experiences. As AR gaming continues to evolve, the possibilities for creative gameplay and storytelling are endless, promising exciting adventures for players to explore.
  • AR, Flutter, Gaming, Machine Learning, Mobile App Development, Software development
  • April 11, 2024
About Author
Subscribe to our newsletter.

Related Articles