Blog

iris

New Android SDK 1.8.0

Posted by iris on April 16, 2019

Hi,

We are glad to let you know that ConnectyCube Android SDK 1.8.0 has beed released!
Here is the change log:

Chat module

Updated

XMPP Smack lib upgrade to 4.3.2

In ConnectycubeRosterEntry:

removed getStatus(), now use isSubscriptionPending() instead.

Fixed

In ConnectycubeRestChatService:

parsing Successfully_deleted ids field when deleteDialogs()

New SDK is available on our Github page.

Stay tuned!

Charlie

ReactNative video chat code sample

Posted by Charlie on April 8, 2019

Hi,

We are happy to introduce our ReactNative video chat code sample, which you can use with your applications as basis or example of correct logic.

Basic UI looks the following way:

Code sample and basic instructions are available on our GitHub page.

Feel free to check ConnectyCube React Native getting started and Video Chat API documentation to ease the development.

Stay tuned! More features are coming!

iris

How to Add Snapchat-like Video Filters with ConnectyCube

Posted by iris on April 2, 2019

Privacy becomes very popular nowadays.
People want to protect their private life and stay anonymous in many cases.
In some chats people would like to stay incognito or apply any other filters to their faces just for fun.

ConnectyCube SDK provides an interface to access video frames so then we can apply OpenCV filters/mask on it and have something like this in 1-1 chat:

Putting mask on face

Wondering how it can be implemented?

Read on!

Android

The following things need to be done on Android platform

1. Connect OpenCV SDK
Import OpenCV Module into your Android project according to A Beginner’s Guide to Setting up OpenCV Android Library on Android Studio

 

2. In ConnectyCube SDK you need to implement a callback on recipient’s side to access video frames, so then we can apply any mask on it:

        videoTrack.addRenderer(new VideoSink() {
            @Override
            public void onFrame(VideoFrame videoFrame) {
                     // apply OpenCV code here 
        // …

                     // render result on a view
      // RTCSurfaceView.onFrame(videoFrame);
            }
        });


More info on ConnectyCube VideoChat API is here

3. Detect face and apply a mask on it with openCV as follows:

Mat putMask(Mat src, Point center, Size face_size){

//mask : masque chargé depuis l'image
Mat mask_resized = new Mat(); //masque resizé
src_roi = new Mat(); //ROI du visage croppé depuis la preview
roi_gray = new Mat();


Imgproc.resize(mask ,mask_resized,face_size);

// ROI selection
roi = new Rect((int) (center.x - face_size.width/2), (int) (center.y - face_size.height/2),(int) face_size.width, (int) face_size.height);
//Rect roi = new Rect(10, 10, (int) face_size.width, (int) face_size.height);

src.submat(roi).copyTo(src_roi);

Log.e(TAG, "MASK SRC1 :"+ src_roi.size());

// to make the white region transparent
Mat mask_grey = new Mat(); //greymask
roi_rgb = new Mat();
Imgproc.cvtColor(mask_resized,mask_grey, Imgproc.COLOR_BGRA2GRAY);
Imgproc.threshold(mask_grey,mask_grey,230,255, Imgproc.THRESH_BINARY_INV);

ArrayList maskChannels = new ArrayList<>(4);
ArrayList result_mask = new ArrayList<>(4);
result_mask.add(new Mat());
result_mask.add(new Mat());
result_mask.add(new Mat());
result_mask.add(new Mat());

Core.split(mask_resized, maskChannels);

Core.bitwise_and(maskChannels.get(0),mask_grey, result_mask.get(0));
Core.bitwise_and(maskChannels.get(1),mask_grey, result_mask.get(1));
Core.bitwise_and(maskChannels.get(2),mask_grey, result_mask.get(2));
Core.bitwise_and(maskChannels.get(3),mask_grey, result_mask.get(3));

Core.merge(result_mask, roi_gray);        

Core.bitwise_not(mask_grey,mask_grey);

ArrayList srcChannels = new ArrayList<>(4);
Core.split(src_roi, srcChannels);
Core.bitwise_and(srcChannels.get(0),mask_grey, result_mask.get(0));
Core.bitwise_and(srcChannels.get(1),mask_grey, result_mask.get(1));
Core.bitwise_and(srcChannels.get(2),mask_grey, result_mask.get(2));
Core.bitwise_and(srcChannels.get(3),mask_grey, result_mask.get(3));

Core.merge(result_mask, roi_rgb);          

Core.addWeighted(roi_gray,1, roi_rgb,1,0, roi_rgb); 

roi_rgb.copyTo(new Mat(src,roi));

return src;}


Read more here.

Note:

All the above is implemented on a recipient’s side, so a sender has to notify the recipient that he/she enabled a mask on their side. You can set such notification via ConnectyCube System Messages API.

 

iOS

The same flow is applied to iOS – ConnectyCube iOS SDK also provides an access to video frames via CMSampleBufferRef.

1. Connect OpenCV SDK
Import OpenCV Module into your iOS project according to this guide.

2. Convert CMSampleBufferRef to cv::Mat: according to the suggestions here.  

- (cv::Mat)matFromBuffer:(CMSampleBufferRef)buffer {
    CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(buffer);
    CVPixelBufferLockBaseAddress( pixelBuffer, 0 );

    //Processing here
    int bufferWidth = (int)CVPixelBufferGetWidth(pixelBuffer);
    int bufferHeight = (int)CVPixelBufferGetHeight(pixelBuffer);
    unsigned char *pixel = (unsigned char *)CVPixelBufferGetBaseAddress(pixelBuffer);

    //put buffer in open cv, no memory copied
    cv::Mat mat = cv::Mat(bufferHeight,bufferWidth,CV_8UC4,pixel,CVPixelBufferGetBytesPerRow(pixelBuffer));

    //End processing
    CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );

    cv::Mat matGray;
    cvtColor(mat, matGray, CV_BGR2GRAY);

    return matGray;
}

 

3. Detect face with openCV according to this guide.

4. Apply a mask on face with OpenCV according to suggestions here.

 

Note:

All the above is implemented on a recipient’s side, so a sender has to notify the recipient that he/she enabled a mask on their side. You can set such notification via ConnectyCube System Messages API.

 

Good luck!

Have any questions or need help?

Feel free to contact us.

iris

PWA for progressive ConnectyCube platform

Posted by iris on March 5, 2019

Progressive Web Apps (PWA) become popular currently.

Google Play supports Progressive Web Apps already.

Apple iOS 12.2 brings more things for PWA support like a maintainable app state.

 

ConnectyCube is also going to be a part of the solution and adjust its JS SDK to comply with PWA requirements.

 

What does PWA mean anyway?

It means that if you have a website which meets certain requirements, you can transform it easily to a mobile app.

You can re-use your code and with minimal changes build an Android or iOS app from it.

Users will not need to open a browser to access it. PWA can have an icon like native standalone apps do.

Also if the user was previously logged in your application via browser on his device, upon installing the PWA app he should be logged in automatically.

 

What does a Web app need to be a Progressive Web App?

Here is the list of requirements to be met:
– responsive UI
– cross-browser support
– offline mode support (data cache, app URLs load while offline)

 

We are going to improve our JS SDK to meet PWA requirements.

Stay tuned!