Mingmin Xiaoxiao from Aufei Temple
Qubit Report丨Public Account QbitAI

Finally, I A childhood dream has come true!

Just need me to take a picture of mine Handwriting, AI can help me transcribe English homework, the kind of "exactly the same" painting style:

I didn’t even copy homework for others Question...

Simply hanging a batch can only An "artifact of homework" that imitates handwriting and costs hundreds or thousands of dollars.

Ahem, focus:

Although the function is very powerful , but this is not for copying English homework for you. (You have to do your homework seriously!)

This is< /span>Facebook AIThe latest "Text Style Brush"(TextStyleBrush), it only needs a photo of handwriting to perfectly restore a whole set of text handwriting.

Not only can move flowers and plants, but also " "soy sauce bottle" becomes "teapot":

It can also be implemented directlyStyle replacement, so that all printed words in the fruit and vegetable shop become handwritten :

In this way, even now Photo text is not necessarily real anymore.

Stronger than format brush: text can also be changed

In actual use, TextStyleBrush is really a Format brush, wherever you need to brush.

Its real power is to simulate handwriting font.

Just enter some text, Add your handwriting, just 1 word, and it can generate a "handwritten version".

This effect can be seen with naked eyes What's more, you can't tell the truth from the fake!

Print the price tag in the market In the process of changing all fonts into handwritten, it can also recognize samples that are not printed, and automatically skip the conversion and synthesis.

△The two handwritten labels have not been changed

When simulating a specific font format, TextStyleBrush also performs very well.

Including posters, trash cans, street signs , beverage bottles, storefront decorations... various text styles can be handled:

In addition to intuitive effects, developers Data analysis was also done on the synthesized images.

The image generated by TextStyleBrush is synthetically wrong (MSE) is greatly reduced, and the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are also improved a lot.

On the accuracy of text recognition , TextStyleBrush performed well in the three sets of data sets:

The accuracy rate is as high as95% above.

Using GAN to change it, it is difficult to distinguish true and false text

According to Facebook, TextStyleBrush is a text style brush based on< span style="color: #00997F; --tt-darkmode-color: #00997F;">Self-supervisedThe model trained by the method can perform style conversion on text with the same text content, just like The format brush is the same.

Of course, not just the format of Word Brush, it can even directly replace the text in the photo, so the model also needs to learntext Methods for identifying and image segmentation.

△Backlight scene is also easy

In order to achieve image segmentation and text style conversion at the same time, the TextStyleBrush model is based onStyleGAN2 designed to generate very photorealistic images.

However, there are two problems with StyleGAN2 :

• First of all, the way it generates images is "randomly hitting", that is, there is no way to control the characteristics of the output image. But TextStyleBrush must generate an image of specified text.
• Secondly, the overall style of StyleGAN2 Uncontrolled, but the style in TextStyleBrush involves a large amount of information combination, including features such as color, scale and style conversion, and even handwriting details with personal characteristics.

To this end, TextStyleBrush firstly controls the output of the model by using text information and style as two "additional conditions" to solve the problem of random image generation by the model.

Then, to further refine To control the style characteristics of the text, various style information in the neural network layer will also be extracted, and these information will be injected into the text generator, so as to control the style of the text from various scales (color, overall style, details).

Besides, due to different The image resolution of the region is different, and the generator must also generate text with a similar resolution to the replacement region.

To this end, the model is added In order to control the structure of high and low resolution, the generated text image can match the resolution of the input image.

Like this, before and after replacement There will be no problems with large differences in font clarity:

But unlike photos, text The style is actually morefree, so sometimes the painting style The authenticity is hard to say.

For this, when training , Facebook introduced an innovative self-supervised training method, combining three models of style classification, text recognition (OCR) and GAN to preserve the input style/text content, and then decide which one to replace.

For example, in text recognition, After letting TextStyleBrush generate a text image, the model will use a pre-trained text recognition structure to "judge" the text content of the image and give it a score.

It turns out that training like this The model is really useful.

Netizen: Disgusting the real? I'm really worried...

Synthesizing human faces has been played too much, but it is the first time to synthesize handwriting.

And it really works good!

So, once the TextStyleBrush is released, it will It attracted many people to watch.

Some netizens have already begun to imagine it Used for:

Welcome to< /span>The world of fancy signature!

LeCun also forwarded a wave.

However, it is really hard to see or not to play It was too uncomfortable, and netizens with itchy hands came to ask questions:

TextStyleBrush will be public Is it open for use?

This naturally leads to a Points that will cause controversy:

The handwriting after synthesis is enough to look like real ones, What if it is misused or used maliciously?

Assume that any one's handwriting is It can be synthesized very easily, so what should we do in many occasions that need to be signed?

For example, some netizens said that if Even doctors imitate their "cursive" prescriptions… …

In addition to security and privacy issues Worry, it's not a problem for type designers good news.

After all, all fonts are actually If there is copyright, if it can be easily simulated, wouldn't it bepiracy< /strong>Flying all over the sky, even the author himself can't tell the truth from the fake.

Some netizens said: This is far from The dystopian world where real and fake are a little bit closer...

In this regard, Facebook's CTO made responded:

Because it may be It is used to forge handwriting, so we only publish papers and data sets, and the source code will not be open source .

share research and datasets, and More to prevent text versions of Deepfakes.

What do you think?

TextStyleBrush dataset:
https://github.com/facebookresearch/IMGUR5K-Handwriting-Dataset

Paper URL:
https://scontent-fml2-1.xx.fbcdn.net/v/t39.8562-6/ 10000000_944085403038430_3779849959048683283_n.pdf

— END —

Qubit QbitAI Toutiao signed a contract

Follow us, the first time to learn about cutting-edge technology trends