Search outside the box: How we're making Search more natural and intuitive

For over two decades, we've dedicated ourselves to our mission: to organize the world’s information and make it universally accessible and useful. We started with text search, but over time, we've continued to create more natural and intuitive ways to find information — you can now search what you see with your camera, or ask a question aloud with your voice.

At Search On today, we showed how advancements in artificial intelligence are enabling us to transform our information products yet again. We're going far beyond the search box to create search experiences that work more like our minds, and that are as multidimensional as we are as people.

We envision a world in which you’ll be able to find exactly what you’re looking for by combining images, sounds, text and speech, just like people do naturally. You’ll be able to ask questions, with fewer words — or even none at all — and we’ll still understand exactly what you mean. And you’ll be able to explore information organized in a way that makes sense to you.

We call this making search more natural and intuitive, and we’re on a long-term path to bring this vision to life for people everywhere. To give you an idea of how we’re evolving the future of our information products, here are three highlights from what we showed today at Search On.

Making visual search work more naturally

Cameras have been around for hundreds of years, and they’re usually thought of as a way to preserve memories, or these days, create content. But a camera is also a powerful way to access information and understand the world around you — so much so that your camera is your next keyboard. That’s why in 2017 we introduced Lens, so you can search what you see using your camera or an image. Now, the age of visual search is here — in fact, people use Lens to answer 8 billion questions every month.

We’re making visual search even more natural with multisearch, a completely new way to search using images and text simultaneously, similar to how you might point at something and ask a friend a question about it. We introduced multisearch earlier this year as a beta in the U.S., and at Search On, we announced we’re expanding it to more than 70 languages in the coming months. We’re taking this capability even further with “multisearch near me,” enabling you to take a picture of an unfamiliar item, such as a dish or plant, then find it at a local place nearby, like a restaurant or gardening shop. We will start rolling “multisearch near me” out in English in the U.S. this fall.

Multisearch enables a completely new way to search using images and text simultaneously.

Translating the world around you

One of the most powerful aspects of visual understanding is its ability to break down language barriers. With advancements in AI, we’ve gone beyond translating text to translating pictures. People already use Google to translate text in images over 1 billion times a month, across more than 100 languages — so they can instantly read storefronts, menus, signs and more.

But often, it’s the combination of words plus context, like background images, that bring meaning. We’re now able to blend translated text into the background image thanks to a machine learning technology called Generative Adversarial Networks (GANs). So if you point your camera at a magazine in another language, for example, you’ll now see translated text realistically overlaid onto the pictures underneath.

With the new Lens translation update, you’ll now see translated text realistically overlaid onto the pictures underneath.

Exploring the world with immersive view

Our quest to create more natural and intuitive experiences also extends to helping you explore the real world. Thanks to advancements in computer vision and predictive models, we're completely reimagining what a map can be. This means you’ll see our 2D map evolve into a multi-dimensional view of the real world, one that allows you to experience a place as if you are there.

Just as live traffic in navigation made Google Maps dramatically more helpful, we’re making another significant advancement in mapping by bringing helpful insights — like weather and how busy a place is — to life with immersive view in Google Maps. With this new experience, you can get a feel for a place before you even step foot inside, so you can confidently decide when and where to go.

Say you’re interested in meeting a friend at a restaurant. You can zoom into the neighborhood and restaurant to get a feel for what it might be like at the date and time you plan to meet up, visualizing things like the weather and learning how busy it might be. By fusing our advanced imagery of the world with our predictive models, we can give you a feel for what a place will be like tomorrow, next week, or even next month. We’re expanding the first iteration of this with aerial views of 250 landmarks today, and immersive view will come to five major cities in the coming months, with more on the way.

Immersive view in Google Maps helps you get a feel for a place before you even visit.

These announcements, along with many others introduced at Search On, are just the start of how we’re transforming our products to help you go beyond the traditional search box. We’re steadfast in our pursuit to create technology that adapts to you and your life — to help you make sense of information in ways that are most natural to you.

by Prabhakar Raghavan via The Keyword